trymirai/Qwen3.5-4B-M
Run locally on Apple devices with Mirai
- Type
- local
- From
- Alibaba
- Quantization
- Mirai Medium
- Parameters
- 4B
- Size
- 2.3 GB
- Source
Mirai's Qwen3.5-4B Medium Quantization
A state-of-the-art quantization that runs at about 84 tok/s on Apple M4 Pro while staying under 2.53 GB resident memory. Read more in our blog post.
Quickstart
If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:
brew install mirai
mirai --model trymirai/Qwen3.5-4B-MCurrently only Apple silicon inference is supported. If you want to build things from source, read this overview.
Method
Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points and bf16 scales, plus block-diagonal Random Hadamard Transforms with block size 32 to suppress outliers. The checkpoint is produced with Post-Training Quantization, using YAQA with additional stabilization heuristics, followed by a lightweight Quantization-Aware Distillation step for further quality recovery.
Benchmarks & Evaluations
In our reasoning-enabled MMLU-Pro evaluation with an 81,920-token limit, Mirai-M reaches 75.9% accuracy, compared with 77.7% for the unquantized BF16 Qwen/Qwen3.5-4B baseline under the same setting.
| Device | Generation | Prefill |
|---|---|---|
| Apple A18 Pro | 19 tok/s | 152 tok/s |
| Apple M1 | 21 tok/s | 146 tok/s |
| Apple M2 | 32 tok/s | 208 tok/s |
| Apple M2 Pro | 62 tok/s | 388 tok/s |
| Apple M3 Max | 117 tok/s | 1,375 tok/s |
| Apple M4 | 38 tok/s | 422 tok/s |
| Apple M4 Pro | 84 tok/s | 815 tok/s |
| Apple M4 Max | 146 tok/s | 1,583 tok/s |
Citation
If you find our work helpful, feel free to give us a cite.
@misc{mirai-quant,
title = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
note = {In collaboration with others at Mirai Labs},
month = {June},
year = {2026},
url = {https://trymirai.com/blog/quantization}
}spm https://github.com/trymirai/uzu.git
| 1 | import Uzu |
| 2 | |
| 3 | public func runChat() async throws { |
| 4 | let engineConfig = EngineConfig.create() |
| 5 | let engine = try await Engine.create(config: engineConfig) |
| 6 | |
| 7 | guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-4B-M") else { |
| 8 | return |
| 9 | } |
| 10 | for try await update in try await engine.download(model: model).iterator() { |
| 11 | print("Download progress: \(update.progress())") |
| 12 | } |
| 13 | |
| 14 | let messages = [ |
| 15 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 16 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 17 | ] |
| 18 | let session = try await engine.chat(model: model, config: .create()) |
| 19 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 20 | var message: ChatMessage? = nil |
| 21 | for try await update in stream.iterator() { |
| 22 | switch update { |
| 23 | case .replies(let replies): |
| 24 | message = replies.last?.message |
| 25 | case .error(let error): |
| 26 | print("Error: \(error)") |
| 27 | } |
| 28 | } |
| 29 | print("Text: \(message?.text() ?? "empty")") |
| 30 | } |