Qwen 3.5 9B
trymirai/Qwen3.5-9B-M on Hugging Face- Vendor
- Alibaba
- Quantization
- Mirai-M
- Parameters
- 9B
- Size
- 6.0 GB
$ brew install mirai$ mirai --model trymirai/Qwen3.5-9B-MBenchmarks on Chip: Apple M5 Max, 18-core CPU, 40-core GPU. OS: macOS 27.0. Model identifier: Mac17,14
Qwen 3.5 9B Mirai-M
Apple M5 Max 128GB
273 tok/s
higher is better ↑
uzu0.6.9vs
0.32.4vsMTPLX2.12.2vsllama.cpp0.7.0vsoMLX0.7.0Benchmarked 20 Oct 2026
Qwen 3.5 9B Mirai-M
Apple M5 Max 128GB
4895 tok/s
higher is better ↑
uzu0.6.9vs0.32.4vsMTPLX2.12.2vsllama.cpp0.7.0vsoMLX0.7.0Benchmarked 20 Oct 2026
Qwen 3.5 9B Mirai-M
Apple M5 Max 128GB
6.12 GB
lower is better ↓
uzu0.6.9vs0.32.4vsMTPLX2.12.2vsllama.cpp0.7.0vsoMLX0.7.0Benchmarked 20 Oct 2026
Mirai Quantization
Speculative decoding
Tokens / secondHigher is better ↑
The performance of speculative decoding heavily depends on the domain: code is easy to predict, whereas general chat queries are much harder. We evaluate it on Mirai’s dataset and report the average generation speed across its samples. This chart shows per-dataset TPS averages.
Benchmarked 20 Oct 2026
Integrate with SDK
1Choose framework
2Run the following command to install Mirai SDK
https://github.com/trymirai/uzu
3Apply code
| 1 | import Foundation |
| 2 | import Uzu |
| 3 | |
| 4 | public func runChat() async throws { |
| 5 | let engineConfig = EngineConfig.create() |
| 6 | let engine = try await Engine.create(config: engineConfig) |
| 7 | |
| 8 | guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-9B-M") else { |
| 9 | return |
| 10 | } |
| 11 | for try await update in try await engine.download(model: model).iterator() { |
| 12 | print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "") |
| 13 | fflush(stdout) |
| 14 | } |
| 15 | print() |
| 16 | |
| 17 | let messages = [ |
| 18 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 19 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 20 | ] |
| 21 | let session = try await engine.chat(model: model, config: .create()) |
| 22 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 23 | var message: ChatMessage? = nil |
| 24 | for try await update in stream.iterator() { |
| 25 | switch update { |
| 26 | case .replies(let replies): |
| 27 | let reply = replies.last |
| 28 | message = reply?.message |
| 29 | print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)") |
| 30 | case .error(let error): |
| 31 | print("Error: \(error)") |
| 32 | case .toolResults: |
| 33 | break |
| 34 | } |
| 35 | } |
| 36 | print("Reasoning: \(message?.reasoning() ?? "empty")") |
| 37 | print("Text: \(message?.text() ?? "empty")") |
| 38 | } |
| 39 | |
TypeScript
pnpm add typescript ts-node @types/node -D pnpm add @trymirai/uzu
import {
ChatConfig,
ChatMessage,
ChatReplyConfig,
ChatSessionStreamChunkError,
ChatSessionStreamChunkReplies,
Engine,
EngineConfig
} from '@trymirai/uzu';
async function main() {
let engineConfig = EngineConfig.create();
let engine = await Engine.create(engineConfig);
let model = await engine.model('trymirai/Qwen3.5-9B-M');
if (!model) {
throw new Error('Model not found');
}
for await (const update of await engine.download(model)) {
process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
}
console.log();
let messages = [
ChatMessage.system().withText('You are a helpful assistant'),
ChatMessage.user().withText('Tell me a short, funny story about a robot')
];
let session = await engine.chat(model, ChatConfig.create());
let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
let message: ChatMessage | undefined;
for await (const chunk of stream) {
if (chunk instanceof ChatSessionStreamChunkReplies) {
message = chunk.replies[0]?.message;
console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
} else if (chunk instanceof ChatSessionStreamChunkError) {
console.error('Error: ', chunk.error);
}
}
console.log('Reasoning: ', message?.reasoning);
console.log('Text: ', message?.text);
}
main().catch((error) => {
console.error(error);
});
Python
uv add uzu
import asyncio
from uzu import (
ChatConfig,
ChatMessage,
ChatReplyConfig,
ChatSessionStreamChunk,
Engine,
EngineConfig,
)
async def main() -> None:
engine_config = EngineConfig.create()
engine = await Engine.create(engine_config)
model = await engine.model("trymirai/Qwen3.5-9B-M")
if model is None:
raise RuntimeError("Model not found")
async for update in (await engine.download(model)).iterator():
print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
print()
messages = [
ChatMessage.system().with_text("You are a helpful assistant"),
ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
]
session = await engine.chat(model, ChatConfig.create())
stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
message: ChatMessage | None = None
async for chunk in stream.iterator():
if isinstance(chunk, ChatSessionStreamChunk.Replies):
replies = chunk.replies
if replies:
reply = replies[0]
message = reply.message
print(f"Generated tokens: {reply.stats.tokens_count_output}")
elif isinstance(chunk, ChatSessionStreamChunk.Error):
print(f"Error: {chunk.error}")
if message is not None:
print(f"Reasoning: {message.reasoning}")
print(f"Text: {message.text}")
if __name__ == "__main__":
asyncio.run(main())
Rust
cargo add uzu --git https://github.com/trymirai/uzu cargo add tokio --features full
use std::io::{self, Write};
use uzu::{
engine::{Engine, EngineConfig},
session::chat::ChatSessionStreamChunk,
types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let engine_config = EngineConfig::default();
let engine = Engine::new(engine_config).await?;
let model = engine.model("trymirai/Qwen3.5-9B-M".to_string()).await?.ok_or("Model not found")?;
let downloader = engine.download(&model).await?;
while let Some(update) = downloader.next().await {
print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
io::stdout().flush()?;
}
println!();
let messages = vec![
ChatMessage::system().with_text("You are a helpful assistant".to_string()),
ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
];
let session = engine.chat(model, ChatConfig::default()).await?;
let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
let mut last_message: Option<ChatMessage> = None;
while let Some(chunk) = stream.next().await {
match chunk {
ChatSessionStreamChunk::ToolResults {
..
} => {},
ChatSessionStreamChunk::Replies {
replies,
} => {
if let Some(reply) = replies.first() {
last_message = Some(reply.message.clone());
println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
}
},
ChatSessionStreamChunk::Error {
error,
} => {
println!("Error: {error}");
},
}
}
if let Some(message) = last_message {
println!("Reasoning: {}", message.reasoning().unwrap_or_default());
println!("Text: {}", message.text().unwrap_or_default());
}
Ok(())
}
Details
Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.
Quickstart
If you are on macOS, the easiest way is to install the mirai Homebrew package and then run the CLI:
sh
brew install mirai
mirai --model trymirai/Qwen3.5-9B-MCurrently only Apple silicon inference is supported. If you want to build things from source, read this overview.
Method
Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 32. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.
Citation
If you find our work helpful, feel free to give us a cite.
bibtex
@misc{mirai-quant,
title = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
note = {In collaboration with others at Mirai Labs},
month = {June},
year = {2026},
url = {https://trymirai.com/blog/quantization}
}Original model
This is a quantized version of Qwen/Qwen3.5-9B. For architecture details, intended use, evaluations, and limitations, see the original model card.