# Mirai Labs

> On-device AI stack for Apple Silicon: the uzu inference engine, Mirai-quantized open models, and a local chat app for macOS. Benchmarks against MLX and llama.cpp are published for every uzu release.

Site: https://trymirai.com. Contact: contact@trymirai.com.

## Benchmarks

- [uzu vs MLX vs llama.cpp: Apple Silicon LLM Benchmarks 2026](https://trymirai.com/metrics): uzu 0.5.25 vs MLX 0.32.1 vs llama.cpp on 10 Macs, 24 quantized models: output and input tok/s and memory. Benchmarked 3 Sept 2026. Engines: uzu 0.5.25, MLX 0.32.1, llama.cpp 0.3.0-dev+c1d0e7a00, MTPLX 2.9.0.

## Models

## Qwen 3.5

### Qwen 3.5 0.8B Mirai-M 4-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-5-0-8b-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.5-0.8B-M
- Model identifier in the uzu SDK: `trymirai/Qwen3.5-0.8B-M`
- Registry id: `alibaba:qwen3.5:0.8b:mirai:mirai-m:4`
- Mirai-M 4-bit, 800M parameters, 460 MB, 262144 context
- Tool calls: no; reasoning: yes


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.5-0.8B-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 32. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.5-0.8B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-0.8B-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.5-0.8B-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.5-0.8B-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.5-0.8B-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 97 t/s | - | 886 t/s | 0.63 GB |
| Apple M1 16GB | MLX 0.32.1 | 80 t/s | - | 847 t/s | 1.40 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 54 t/s | - | 1106 t/s | 0.60 GB |
| Apple M2 24GB | uzu 0.5.25 | 132 t/s | - | 1258 t/s | 0.63 GB |
| Apple M2 24GB | MLX 0.32.1 | 114 t/s | - | 1182 t/s | 1.40 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 89 t/s | - | 1586 t/s | 0.60 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 219 t/s | - | 2278 t/s | 0.63 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 192 t/s | - | 1841 t/s | 1.63 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 141 t/s | - | 2898 t/s | 0.60 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 340 t/s | - | 7186 t/s | 0.63 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 319 t/s | - | 2835 t/s | 1.63 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 216 t/s | - | 6327 t/s | 0.60 GB |
| Apple M4 32GB | uzu 0.5.25 | 162 t/s | - | 2528 t/s | 0.63 GB |
| Apple M4 32GB | MLX 0.32.1 | 150 t/s | - | 1874 t/s | 1.40 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 109 t/s | - | 2023 t/s | 0.58 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 288 t/s | - | 4673 t/s | 0.63 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 267 t/s | - | 3057 t/s | 1.63 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 193 t/s | - | 3899 t/s | 0.60 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 405 t/s | - | 8641 t/s | 0.63 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 384 t/s | - | 3160 t/s | 1.63 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 287 t/s | - | 6434 t/s | 0.60 GB |
| Apple M5 32GB | uzu 0.5.25 | 215 t/s | - | 5558 t/s | 0.67 GB |
| Apple M5 32GB | MLX 0.32.1 | 182 t/s | - | 3508 t/s | 1.40 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 146 t/s | - | 2306 t/s | 0.57 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 370 t/s | - | 9997 t/s | 0.67 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 236 t/s | - | 3182 t/s | 1.63 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 169 t/s | - | 5459 t/s | 0.77 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 519 t/s | - | 17097 t/s | 0.67 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 443 t/s | - | 4820 t/s | 1.63 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 302 t/s | - | 13023 t/s | 0.77 GB |

### Qwen 3.5 0.8B Mirai-L 8-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-5-0-8b-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.5-0.8B-L
- Model identifier in the uzu SDK: `trymirai/Qwen3.5-0.8B-L`
- Registry id: `alibaba:qwen3.5:0.8b:mirai:mirai-l:8`
- Mirai-L 8-bit, 800M parameters, 824 MB, 262144 context
- Tool calls: no; reasoning: yes


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.5-0.8B-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 32. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.5-0.8B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-0.8B-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.5-0.8B-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.5-0.8B-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.5-0.8B-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 60 t/s | - | 1109 t/s | 0.96 GB |
| Apple M1 16GB | MLX 0.32.1 | 54 t/s | - | 845 t/s | 1.75 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 35 t/s | - | 1231 t/s | 1.23 GB |
| Apple M2 24GB | uzu 0.5.25 | 85 t/s | - | 1578 t/s | 0.96 GB |
| Apple M2 24GB | MLX 0.32.1 | 79 t/s | - | 1189 t/s | 1.75 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 55 t/s | - | 1735 t/s | 1.23 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 158 t/s | - | 2825 t/s | 0.96 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 145 t/s | - | 1853 t/s | 1.95 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 90 t/s | - | 3226 t/s | 1.23 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 266 t/s | - | 7198 t/s | 0.96 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 246 t/s | - | 3573 t/s | 1.95 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 163 t/s | - | 6669 t/s | 1.23 GB |
| Apple M4 32GB | uzu 0.5.25 | 107 t/s | - | 2539 t/s | 0.96 GB |
| Apple M4 32GB | MLX 0.32.1 | 98 t/s | - | 1856 t/s | 1.75 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 65 t/s | - | 2102 t/s | 1.16 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 216 t/s | - | 4694 t/s | 0.96 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 193 t/s | - | 2957 t/s | 1.95 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 127 t/s | - | 4110 t/s | 1.23 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 332 t/s | - | 8671 t/s | 0.96 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 295 t/s | - | 3597 t/s | 1.95 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 210 t/s | - | 7401 t/s | 1.23 GB |
| Apple M5 32GB | uzu 0.5.25 | 133 t/s | - | 6485 t/s | 1.00 GB |
| Apple M5 32GB | MLX 0.32.1 | 113 t/s | - | 3334 t/s | 1.75 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 83 t/s | - | 2363 t/s | 1.21 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 232 t/s | - | 11332 t/s | 1.00 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 169 t/s | - | 3880 t/s | 1.95 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 116 t/s | - | 6292 t/s | 1.41 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 392 t/s | - | 19315 t/s | 1.00 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 349 t/s | - | 6471 t/s | 1.95 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 227 t/s | - | 13094 t/s | 1.40 GB |

### Qwen 3.5 2B Mirai-M 4-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-5-2b-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.5-2B-M
- Model identifier in the uzu SDK: `trymirai/Qwen3.5-2B-M`
- Registry id: `alibaba:qwen3.5:2b:mirai:mirai-m:4`
- Mirai-M 4-bit, 2B parameters, 1.1 GB, 262144 context
- Tool calls: no; reasoning: yes


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.5-2B-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 32. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.5-2B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-2B-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.5-2B-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.5-2B-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.5-2B-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 47 t/s | - | 376 t/s | 1.23 GB |
| Apple M1 16GB | MLX 0.32.1 | 43 t/s | - | 406 t/s | 1.94 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 34 t/s | - | 508 t/s | 1.27 GB |
| Apple M2 24GB | uzu 0.5.25 | 67 t/s | - | 535 t/s | 1.23 GB |
| Apple M2 24GB | MLX 0.32.1 | 63 t/s | - | 570 t/s | 1.94 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 52 t/s | - | 726 t/s | 1.23 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 121 t/s | - | 993 t/s | 1.23 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 114 t/s | - | 985 t/s | 2.15 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 87 t/s | - | 1351 t/s | 1.27 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 211 t/s | - | 3384 t/s | 1.23 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 209 t/s | - | 2327 t/s | 2.15 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 156 t/s | - | 3168 t/s | 1.30 GB |
| Apple M4 32GB | uzu 0.5.25 | 81 t/s | - | 1069 t/s | 1.23 GB |
| Apple M4 32GB | MLX 0.32.1 | 78 t/s | - | 958 t/s | 1.94 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 62 t/s | - | 947 t/s | 1.27 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 158 t/s | - | 2058 t/s | 1.23 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 157 t/s | - | 1648 t/s | 2.15 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 121 t/s | - | 1873 t/s | 1.27 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 256 t/s | - | 3943 t/s | 1.23 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 253 t/s | - | 2821 t/s | 2.15 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 194 t/s | - | 3641 t/s | 1.26 GB |
| Apple M5 32GB | uzu 0.5.25 | 100 t/s | - | 2703 t/s | 1.21 GB |
| Apple M5 32GB | MLX 0.32.1 | 89 t/s | - | 2322 t/s | 1.94 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 78 t/s | - | 1045 t/s | 1.15 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 180 t/s | - | 4722 t/s | 1.21 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 138 t/s | - | 2958 t/s | 2.15 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 114 t/s | - | 3675 t/s | 1.44 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 319 t/s | - | 8982 t/s | 1.21 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 295 t/s | - | 5037 t/s | 2.15 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 221 t/s | - | 6892 t/s | 1.43 GB |

### Qwen 3.5 2B Mirai-L 8-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-5-2b-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.5-2B-L
- Model identifier in the uzu SDK: `trymirai/Qwen3.5-2B-L`
- Registry id: `alibaba:qwen3.5:2b:mirai:mirai-l:8`
- Mirai-L 8-bit, 2B parameters, 2.0 GB, 262144 context
- Tool calls: no; reasoning: yes


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.5-2B-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 32. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.5-2B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-2B-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.5-2B-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.5-2B-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.5-2B-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 27 t/s | - | 483 t/s | 2.08 GB |
| Apple M1 16GB | MLX 0.32.1 | 26 t/s | - | 407 t/s | 2.82 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 18 t/s | - | 560 t/s | 2.74 GB |
| Apple M2 24GB | uzu 0.5.25 | 41 t/s | - | 688 t/s | 2.08 GB |
| Apple M2 24GB | MLX 0.32.1 | 39 t/s | - | 577 t/s | 2.82 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 28 t/s | - | 818 t/s | 2.75 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 76 t/s | - | 1275 t/s | 2.08 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 73 t/s | - | 994 t/s | 2.83 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 51 t/s | - | 1536 t/s | 2.75 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 145 t/s | - | 3409 t/s | 2.08 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 140 t/s | - | 2335 t/s | 2.83 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 97 t/s | - | 3274 t/s | 2.74 GB |
| Apple M4 32GB | uzu 0.5.25 | 47 t/s | - | 1078 t/s | 2.08 GB |
| Apple M4 32GB | MLX 0.32.1 | 46 t/s | - | 949 t/s | 2.82 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 31 t/s | - | 986 t/s | 2.75 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 105 t/s | - | 2075 t/s | 2.08 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 100 t/s | - | 1637 t/s | 2.83 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 70 t/s | - | 1963 t/s | 2.75 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 182 t/s | - | 3981 t/s | 2.08 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 169 t/s | - | 2784 t/s | 2.83 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 123 t/s | - | 3821 t/s | 2.77 GB |
| Apple M5 32GB | uzu 0.5.25 | 58 t/s | - | 3212 t/s | 2.06 GB |
| Apple M5 32GB | MLX 0.32.1 | 54 t/s | - | 2252 t/s | 2.82 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 41 t/s | - | 1074 t/s | 2.74 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 124 t/s | - | 6124 t/s | 2.06 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 87 t/s | - | 2923 t/s | 2.83 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 61 t/s | - | 3720 t/s | 2.95 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 216 t/s | - | 10372 t/s | 2.06 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 199 t/s | - | 4997 t/s | 2.83 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 140 t/s | - | 8420 t/s | 2.94 GB |

### Qwen 3.5 4B Mirai-M 4-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-5-4b-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.5-4B-M
- Model identifier in the uzu SDK: `trymirai/Qwen3.5-4B-M`
- Registry id: `alibaba:qwen3.5:4b:mirai:mirai-m:4`
- Mirai-M 4-bit, 4B parameters, 2.5 GB, 262144 context
- Tool calls: no; reasoning: yes


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.5-4B-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 32. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-4B-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.5-4B-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.5-4B-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.5-4B-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 21 t/s | - | 156 t/s | 2.65 GB |
| Apple M1 16GB | MLX 0.32.1 | 21 t/s | - | 166 t/s | 3.42 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 17 t/s | - | 198 t/s | 2.69 GB |
| Apple M2 24GB | uzu 0.5.25 | 31 t/s | - | 222 t/s | 2.65 GB |
| Apple M2 24GB | MLX 0.32.1 | 30 t/s | - | 233 t/s | 3.42 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 24 t/s | - | 286 t/s | 2.70 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 58 t/s | - | 415 t/s | 2.65 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 59 t/s | - | 428 t/s | 3.58 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 41 t/s | - | 537 t/s | 2.71 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 106 t/s | - | 1369 t/s | 2.65 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 111 t/s | - | 1114 t/s | 3.58 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 81 t/s | - | 1240 t/s | 2.69 GB |
| Apple M4 32GB | uzu 0.5.25 | 37 t/s | - | 422 t/s | 2.65 GB |
| Apple M4 32GB | MLX 0.32.1 | 37 t/s | - | 397 t/s | 3.42 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 30 t/s | - | 373 t/s | 2.70 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 78 t/s | - | 819 t/s | 2.65 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 79 t/s | - | 744 t/s | 3.58 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 59 t/s | - | 740 t/s | 2.71 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 128 t/s | - | 1580 t/s | 2.65 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 134 t/s | - | 1379 t/s | 3.58 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 102 t/s | - | 1443 t/s | 2.70 GB |
| Apple M5 32GB | uzu 0.5.25 | 45 t/s | - | 1140 t/s | 2.66 GB |
| Apple M5 32GB | MLX 0.32.1 | 44 t/s | - | 1149 t/s | 3.42 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 38 t/s | - | 412 t/s | 2.69 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 96 t/s | - | 2139 t/s | 2.66 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 69 t/s | - | 1573 t/s | 3.58 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 58 t/s | - | 1683 t/s | 2.95 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 159 t/s | - | 3894 t/s | 2.66 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 157 t/s | - | 3284 t/s | 3.58 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 124 t/s | - | 3693 t/s | 2.94 GB |

### Qwen 3.5 4B Mirai-L 8-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-5-4b-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.5-4B-L
- Model identifier in the uzu SDK: `trymirai/Qwen3.5-4B-L`
- Registry id: `alibaba:qwen3.5:4b:mirai:mirai-l:8`
- Mirai-L 8-bit, 4B parameters, 4.5 GB, 262144 context
- Tool calls: no; reasoning: yes


*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.5-4B-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 32. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.5-4B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-4B-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.5-4B-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.5-4B-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.5-4B-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 12 t/s | - | 195 t/s | 4.56 GB |
| Apple M1 16GB | MLX 0.32.1 | 12 t/s | - | 163 t/s | 5.30 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 9 t/s | - | 217 t/s | 5.84 GB |
| Apple M2 24GB | uzu 0.5.25 | 18 t/s | - | 278 t/s | 4.56 GB |
| Apple M2 24GB | MLX 0.32.1 | 18 t/s | - | 236 t/s | 5.30 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 13 t/s | - | 325 t/s | 5.83 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 36 t/s | - | 521 t/s | 4.56 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 35 t/s | - | 429 t/s | 5.46 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 25 t/s | - | 617 t/s | 5.83 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 70 t/s | - | 1369 t/s | 4.56 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 68 t/s | - | 1139 t/s | 5.46 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 49 t/s | - | 1313 t/s | 5.82 GB |
| Apple M4 32GB | uzu 0.5.25 | 21 t/s | - | 422 t/s | 4.56 GB |
| Apple M4 32GB | MLX 0.32.1 | 21 t/s | - | 392 t/s | 5.30 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 16 t/s | - | 391 t/s | 5.83 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 49 t/s | - | 819 t/s | 4.56 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 47 t/s | - | 733 t/s | 5.46 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 34 t/s | - | 776 t/s | 5.84 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 87 t/s | - | 1581 t/s | 4.56 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 85 t/s | - | 1354 t/s | 5.46 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 62 t/s | - | 1522 t/s | 5.83 GB |
| Apple M5 32GB | uzu 0.5.25 | 26 t/s | - | 1359 t/s | 4.55 GB |
| Apple M5 32GB | MLX 0.32.1 | 25 t/s | - | 1050 t/s | 5.30 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 20 t/s | - | 431 t/s | 5.82 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 57 t/s | - | 2597 t/s | 4.55 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 41 t/s | - | 1527 t/s | 5.46 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 30 t/s | - | 1615 t/s | 6.01 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 101 t/s | - | 4773 t/s | 4.55 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 95 t/s | - | 3101 t/s | 5.46 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 72 t/s | - | 3762 t/s | 6.07 GB |

### Qwen 3.5 9B Mirai-M 4-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-5-9b-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.5-9B-M
- Model identifier in the uzu SDK: `trymirai/Qwen3.5-9B-M`
- Registry id: `alibaba:qwen3.5:9b:mirai:mirai-m:4`
- Mirai-M 4-bit, 9B parameters, 5.2 GB, 262144 context
- Tool calls: no; reasoning: yes


*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.5-9B-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 32. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-9B-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.5-9B-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.5-9B-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.5-9B-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 12 t/s | - | 84 t/s | 5.15 GB |
| Apple M1 16GB | MLX 0.32.1 | 12 t/s | - | 91 t/s | 5.86 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 10 t/s | - | 108 t/s | 5.30 GB |
| Apple M2 24GB | uzu 0.5.25 | 18 t/s | - | 119 t/s | 5.15 GB |
| Apple M2 24GB | MLX 0.32.1 | 18 t/s | - | 129 t/s | 5.86 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 15 t/s | - | 158 t/s | 5.31 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 35 t/s | - | 225 t/s | 5.15 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 35 t/s | - | 241 t/s | 6.03 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 27 t/s | - | 299 t/s | 5.31 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 65 t/s | - | 756 t/s | 5.15 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 68 t/s | - | 696 t/s | 6.03 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 55 t/s | - | 709 t/s | 5.30 GB |
| Apple M4 32GB | uzu 0.5.25 | 21 t/s | - | 230 t/s | 5.15 GB |
| Apple M4 32GB | MLX 0.32.1 | 21 t/s | - | 223 t/s | 5.86 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 18 t/s | - | 209 t/s | 5.31 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 46 t/s | - | 452 t/s | 5.15 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 48 t/s | - | 429 t/s | 6.03 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 39 t/s | - | 415 t/s | 5.31 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 83 t/s | - | 884 t/s | 5.15 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 85 t/s | - | 814 t/s | 6.03 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 69 t/s | - | 820 t/s | 5.32 GB |
| Apple M5 32GB | uzu 0.5.25 | 25 t/s | - | 630 t/s | 5.21 GB |
| Apple M5 32GB | MLX 0.32.1 | 26 t/s | - | 727 t/s | 5.86 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 23 t/s | - | 230 t/s | 5.30 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 53 t/s | - | 1195 t/s | 5.21 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 51 t/s | - | 1254 t/s | 6.03 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 40 t/s | - | 1072 t/s | 5.56 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 101 t/s | - | 2251 t/s | 5.21 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 71 t/s | - | 2129 t/s | 6.03 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 83 t/s | - | 2264 t/s | 5.52 GB |

### Qwen 3.5 9B Mirai-L 8-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-5-9b-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.5-9B-L
- Model identifier in the uzu SDK: `trymirai/Qwen3.5-9B-L`
- Registry id: `alibaba:qwen3.5:9b:mirai:mirai-l:8`
- Mirai-L 8-bit, 9B parameters, 9.3 GB, 262144 context
- Tool calls: no; reasoning: yes


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.5-9B-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 64. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.5-9B-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.5-9B-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.5-9B-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.5-9B-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M2 24GB | uzu 0.5.25 | 11 t/s | - | 153 t/s | 8.95 GB |
| Apple M2 24GB | MLX 0.32.1 | 10 t/s | - | 130 t/s | 9.94 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 8 t/s | - | 179 t/s | 12.36 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 21 t/s | - | 291 t/s | 8.95 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 20 t/s | - | 241 t/s | 10.11 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 15 t/s | - | 346 t/s | 12.36 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 41 t/s | - | 760 t/s | 8.95 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 41 t/s | - | 682 t/s | 10.11 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 30 t/s | - | 748 t/s | 12.36 GB |
| Apple M4 32GB | uzu 0.5.25 | 12 t/s | - | 232 t/s | 8.95 GB |
| Apple M4 32GB | MLX 0.32.1 | 12 t/s | - | 219 t/s | 9.94 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 9 t/s | - | 218 t/s | 12.35 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 29 t/s | - | 455 t/s | 8.95 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 27 t/s | - | 423 t/s | 10.11 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 21 t/s | - | 436 t/s | 12.36 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 53 t/s | - | 892 t/s | 8.95 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 51 t/s | - | 797 t/s | 10.11 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 39 t/s | - | 864 t/s | 12.38 GB |
| Apple M5 32GB | uzu 0.5.25 | 15 t/s | - | 814 t/s | 8.98 GB |
| Apple M5 32GB | MLX 0.32.1 | 14 t/s | - | 632 t/s | 9.94 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 11 t/s | - | 234 t/s | 12.33 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 32 t/s | - | 1582 t/s | 8.98 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 28 t/s | - | 1141 t/s | 10.11 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 22 t/s | - | 1104 t/s | 12.44 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 63 t/s | - | 3099 t/s | 8.98 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 47 t/s | - | 1937 t/s | 10.11 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 45 t/s | - | 2258 t/s | 12.50 GB |

## Qwen 3.6

### Qwen 3.6 27B Mirai-M 4-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-6-27b-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.6-27B-M
- Model identifier in the uzu SDK: `trymirai/Qwen3.6-27B-M`
- Registry id: `alibaba:qwen3.6:27b:mirai:mirai-m:4`
- Mirai-M 4-bit, 27B parameters, 15.6 GB, 262144 context
- Tool calls: no; reasoning: yes


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.6-27B-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 64. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.

In quantization and model-size tier, this checkpoint is comparable to Unsloth's [`Q4_K_M`](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/blob/main/Qwen3.6-27B-Q4_K_M.gguf) GGUF.

#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.6-27B-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.6-27B-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.6-27B-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.6-27B-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M2 24GB | uzu 0.5.25 | 6 t/s | - | 34 t/s | 14.40 GB |
| Apple M2 24GB | MLX 0.32.1 | 6 t/s | - | 36 t/s | 16.05 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 5 t/s | - | 45 t/s | 15.45 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 12 t/s | - | 64 t/s | 14.40 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 11 t/s | - | 66 t/s | 16.25 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 9 t/s | - | 85 t/s | 15.61 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 23 t/s | - | 219 t/s | 14.40 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 23 t/s | - | 213 t/s | 16.25 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 18 t/s | - | 197 t/s | 15.65 GB |
| Apple M4 32GB | uzu 0.5.25 | 7 t/s | - | 65 t/s | 14.40 GB |
| Apple M4 32GB | MLX 0.32.1 | 7 t/s | - | 61 t/s | 16.05 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 6 t/s | - | 60 t/s | 15.58 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 16 t/s | - | 128 t/s | 14.40 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 15 t/s | - | 126 t/s | 16.25 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 13 t/s | - | 117 t/s | 15.65 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 29 t/s | - | 252 t/s | 14.40 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 29 t/s | - | 247 t/s | 16.25 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 24 t/s | - | 229 t/s | 15.64 GB |
| Apple M5 32GB | uzu 0.5.25 | 8 t/s | 36 t/s | 216 t/s | 15.39 GB |
| Apple M5 32GB | MLX 0.32.1 | 8 t/s | - | 198 t/s | 16.05 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 7 t/s | 10 t/s | 151 t/s | 16.13 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 18 t/s | 66 t/s | 420 t/s | 15.39 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 14 t/s | - | 360 t/s | 16.25 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 12 t/s | 20 t/s | 288 t/s | 17.13 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 35 t/s | 119 t/s | 834 t/s | 15.39 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 26 t/s | - | 541 t/s | 16.25 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 28 t/s | 30 t/s | 671 t/s | 17.29 GB |

### Qwen 3.6 27B Mirai-L 8-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-6-27b-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.6-27B-L
- Model identifier in the uzu SDK: `trymirai/Qwen3.6-27B-L`
- Registry id: `alibaba:qwen3.6:27b:mirai:mirai-l:8`
- Mirai-L 8-bit, 27B parameters, 28.8 GB, 262144 context
- Tool calls: no; reasoning: yes


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.6-27B-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 64. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.

In quantization and model-size tier, this checkpoint is comparable to Unsloth's [`UD-Q8_K_XL`](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/blob/main/Qwen3.6-27B-UD-Q8_K_XL.gguf) GGUF.

#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.6-27B-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.6-27B-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.6-27B-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.6-27B-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M3 Max 48GB | uzu 0.5.25 | 14 t/s | - | 220 t/s | 26.74 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 13 t/s | - | 148 t/s | 28.71 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 10 t/s | - | 214 t/s | 33.36 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 9 t/s | - | 127 t/s | 26.74 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 8 t/s | - | 115 t/s | 28.71 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 7 t/s | - | 125 t/s | 33.06 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 17 t/s | - | 252 t/s | 26.74 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 16 t/s | - | 240 t/s | 28.71 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 13 t/s | - | 247 t/s | 33.37 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 10 t/s | 45 t/s | 490 t/s | 27.70 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 10 t/s | - | 273 t/s | 28.71 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 8 t/s | 16 t/s | 378 t/s | 35.09 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 20 t/s | 82 t/s | 929 t/s | 27.70 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 16 t/s | - | 487 t/s | 47.20 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 13 t/s | 19 t/s | 557 t/s | 35.16 GB |

## Qwen 3.8

### Qwen 3.8 27B Mirai-M 4-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-8-27b-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.8-27B-M
- Model identifier in the uzu SDK: `trymirai/Qwen3.8-27B-M`
- Registry id: `alibaba:qwen3.8:27b:mirai:mirai-m:4`
- Mirai-M 4-bit, 27B parameters, 15.5 GB, 262144 context
- Tool calls: no; reasoning: yes


*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.8-27B-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 64. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.

In quantization and model-size tier, this checkpoint is comparable to Unsloth's [`UD-Q4_K_M`](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_M.gguf) GGUF.

#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.8-27B-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.8-27B-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.8-27B-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.8-27B-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M2 24GB | uzu 0.5.25 | 6 t/s | - | 34 t/s | 14.40 GB |
| Apple M2 24GB | MLX 0.32.1 | 6 t/s | - | 36 t/s | 16.05 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 5 t/s | - | 45 t/s | 14.63 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 12 t/s | - | 65 t/s | 14.40 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 11 t/s | - | 67 t/s | 16.31 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 9 t/s | - | 87 t/s | 14.61 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 23 t/s | - | 221 t/s | 14.40 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 23 t/s | - | 208 t/s | 16.31 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 19 t/s | - | 204 t/s | 14.58 GB |
| Apple M4 32GB | uzu 0.5.25 | 7 t/s | - | 66 t/s | 14.40 GB |
| Apple M4 32GB | MLX 0.32.1 | 7 t/s | - | 61 t/s | 16.05 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 6 t/s | - | 61 t/s | 14.61 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 16 t/s | - | 129 t/s | 14.40 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 15 t/s | - | 127 t/s | 16.31 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 13 t/s | - | 122 t/s | 14.57 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 29 t/s | - | 256 t/s | 14.40 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 29 t/s | - | 249 t/s | 16.31 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 23 t/s | - | 242 t/s | 14.58 GB |
| Apple M5 32GB | uzu 0.5.25 | 8 t/s | - | 220 t/s | 14.43 GB |
| Apple M5 32GB | MLX 0.32.1 | 8 t/s | - | 204 t/s | 16.05 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 8 t/s | - | 66 t/s | 14.61 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 19 t/s | - | 439 t/s | 14.43 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 14 t/s | - | 349 t/s | 16.31 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 13 t/s | - | 345 t/s | 15.08 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 36 t/s | - | 846 t/s | 14.43 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 29 t/s | - | 626 t/s | 16.31 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 29 t/s | - | 648 t/s | 14.87 GB |

### Qwen 3.8 27B Mirai-L 8-bit

- Page: https://trymirai.com/local-models/alibaba-qwen3-8-27b-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/Qwen3.8-27B-L
- Model identifier in the uzu SDK: `trymirai/Qwen3.8-27B-L`
- Registry id: `alibaba:qwen3.8:27b:mirai:mirai-l:8`
- Mirai-L 8-bit, 27B parameters, 28.8 GB, 262144 context
- Tool calls: no; reasoning: yes


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Qwen3.8-27B-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 64. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.

In quantization and model-size tier, this checkpoint is comparable to Unsloth's [`UD-Q8_K_XL`](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q8_K_XL.gguf) GGUF.

#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Qwen3.8-27B-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Qwen3.8-27B-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Qwen3.8-27B-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Qwen3.8-27B-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M3 Max 48GB | uzu 0.5.25 | 14 t/s | - | 223 t/s | 26.74 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 13 t/s | - | 153 t/s | 28.76 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 11 t/s | - | 218 t/s | 29.36 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 9 t/s | - | 129 t/s | 26.74 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 8 t/s | - | 117 t/s | 28.76 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 8 t/s | - | 126 t/s | 29.36 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 17 t/s | - | 256 t/s | 26.74 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 16 t/s | - | 241 t/s | 28.76 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 15 t/s | - | 252 t/s | 29.36 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 10 t/s | - | 499 t/s | 26.77 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 7 t/s | - | 210 t/s | 28.76 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 7 t/s | - | 306 t/s | 29.43 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 20 t/s | - | 973 t/s | 26.77 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 15 t/s | - | 632 t/s | 28.76 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 17 t/s | - | 738 t/s | 26.13 GB |

## LFM 2.5

### LFM 2.5 230M Mirai-M 4-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-230m-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-230M-M
- Model identifier in the uzu SDK: `trymirai/LFM2.5-230M-M`
- Registry id: `liquidai:lfm2.5:230m:mirai:mirai-m:4`
- Mirai-M 4-bit, 230M parameters, 130 MB, 128000 context
- Tool calls: no; reasoning: no


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-230M-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 64. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-230M/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-230M-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-230M-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-230M-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-230M-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 317 t/s | - | 2794 t/s | 0.21 GB |
| Apple M1 16GB | MLX 0.32.1 | 225 t/s | - | 2708 t/s | 0.65 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 113 t/s | - | 3858 t/s | 0.25 GB |
| Apple M2 24GB | uzu 0.5.25 | 426 t/s | - | 4016 t/s | 0.21 GB |
| Apple M2 24GB | MLX 0.32.1 | 300 t/s | - | 3830 t/s | 0.65 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 281 t/s | - | 5580 t/s | 0.23 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 620 t/s | - | 7406 t/s | 0.21 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 501 t/s | - | 4981 t/s | 0.65 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 409 t/s | - | 10059 t/s | 0.25 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 946 t/s | - | 23902 t/s | 0.21 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 762 t/s | - | 6363 t/s | 0.63 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 505 t/s | - | 17801 t/s | 0.25 GB |
| Apple M4 32GB | uzu 0.5.25 | 540 t/s | - | 8218 t/s | 0.21 GB |
| Apple M4 32GB | MLX 0.32.1 | 380 t/s | - | 5681 t/s | 0.65 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 348 t/s | - | 7123 t/s | 0.23 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 889 t/s | - | 15974 t/s | 0.21 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 664 t/s | - | 7312 t/s | 0.65 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 546 t/s | - | 12604 t/s | 0.22 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 1185 t/s | - | 29969 t/s | 0.21 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 899 t/s | - | 6639 t/s | 0.61 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 704 t/s | - | 17193 t/s | 0.22 GB |
| Apple M5 32GB | uzu 0.5.25 | 677 t/s | - | 17601 t/s | 0.23 GB |
| Apple M5 32GB | MLX 0.32.1 | 447 t/s | - | 8800 t/s | 0.65 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 456 t/s | - | 7483 t/s | 0.25 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 1072 t/s | - | 30284 t/s | 0.23 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 731 t/s | - | 10689 t/s | 0.58 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 452 t/s | - | 23295 t/s | 0.30 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 1392 t/s | - | 44007 t/s | 0.23 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 1014 t/s | - | 8864 t/s | 0.59 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 818 t/s | - | 37706 t/s | 0.30 GB |

### LFM 2.5 230M Mirai-L 8-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-230m-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-230M-L
- Model identifier in the uzu SDK: `trymirai/LFM2.5-230M-L`
- Registry id: `liquidai:lfm2.5:230m:mirai:mirai-l:8`
- Mirai-L 8-bit, 230M parameters, 243 MB, 128000 context
- Tool calls: no; reasoning: no


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-230M-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 64. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-230M/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-230M-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-230M-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-230M-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-230M-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 196 t/s | - | 3787 t/s | 0.31 GB |
| Apple M1 16GB | MLX 0.32.1 | 162 t/s | - | 2880 t/s | 0.74 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 99 t/s | - | 4225 t/s | 0.42 GB |
| Apple M2 24GB | uzu 0.5.25 | 278 t/s | - | 5418 t/s | 0.31 GB |
| Apple M2 24GB | MLX 0.32.1 | 215 t/s | - | 3987 t/s | 0.74 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 176 t/s | - | 6165 t/s | 0.40 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 465 t/s | - | 9977 t/s | 0.31 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 410 t/s | - | 5352 t/s | 0.74 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 284 t/s | - | 10774 t/s | 0.42 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 750 t/s | - | 24031 t/s | 0.31 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 642 t/s | - | 8753 t/s | 0.74 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 403 t/s | - | 20780 t/s | 0.42 GB |
| Apple M4 32GB | uzu 0.5.25 | 351 t/s | - | 8397 t/s | 0.31 GB |
| Apple M4 32GB | MLX 0.32.1 | 272 t/s | - | 6072 t/s | 0.74 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 215 t/s | - | 7447 t/s | 0.42 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 653 t/s | - | 16307 t/s | 0.31 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 530 t/s | - | 7977 t/s | 0.74 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 381 t/s | - | 13845 t/s | 0.42 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 954 t/s | - | 30524 t/s | 0.31 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 763 t/s | - | 9229 t/s | 0.72 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 563 t/s | - | 18009 t/s | 0.42 GB |
| Apple M5 32GB | uzu 0.5.25 | 418 t/s | - | 21692 t/s | 0.35 GB |
| Apple M5 32GB | MLX 0.32.1 | 326 t/s | - | 9020 t/s | 0.74 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 273 t/s | - | 7659 t/s | 0.42 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 771 t/s | - | 33231 t/s | 0.35 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 581 t/s | - | 14047 t/s | 0.70 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 336 t/s | - | 28281 t/s | 0.49 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 1110 t/s | - | 55591 t/s | 0.35 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 866 t/s | - | 12310 t/s | 0.70 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 673 t/s | - | 38508 t/s | 0.48 GB |

### LFM 2.5 350M Mirai-M 4-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-350m-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-350M-M
- Model identifier in the uzu SDK: `trymirai/LFM2.5-350M-M`
- Registry id: `liquidai:lfm2.5:350m:mirai:mirai-m:4`
- Mirai-M 4-bit, 350M parameters, 198 MB, 128000 context
- Tool calls: no; reasoning: no


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-350M-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 64. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-350M/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-350M-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-350M-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-350M-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-350M-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 227 t/s | - | 1888 t/s | 0.30 GB |
| Apple M1 16GB | MLX 0.32.1 | 143 t/s | - | 1952 t/s | 1.63 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 107 t/s | - | 2454 t/s | 0.31 GB |
| Apple M2 24GB | uzu 0.5.25 | 313 t/s | - | 2697 t/s | 0.30 GB |
| Apple M2 24GB | MLX 0.32.1 | 207 t/s | - | 2865 t/s | 1.63 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 237 t/s | - | 3644 t/s | 0.31 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 483 t/s | - | 4973 t/s | 0.30 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 392 t/s | - | 4046 t/s | 1.64 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 374 t/s | - | 6175 t/s | 0.31 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 738 t/s | - | 15179 t/s | 0.30 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 585 t/s | - | 5361 t/s | 1.55 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 441 t/s | - | 12616 t/s | 0.31 GB |
| Apple M4 32GB | uzu 0.5.25 | 382 t/s | - | 5168 t/s | 0.30 GB |
| Apple M4 32GB | MLX 0.32.1 | 236 t/s | - | 3696 t/s | 1.61 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 285 t/s | - | 4511 t/s | 0.31 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 668 t/s | - | 10035 t/s | 0.30 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 456 t/s | - | 5807 t/s | 1.60 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 464 t/s | - | 8472 t/s | 0.32 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 906 t/s | - | 18884 t/s | 0.30 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 673 t/s | - | 7922 t/s | 1.60 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 602 t/s | - | 14043 t/s | 0.31 GB |
| Apple M5 32GB | uzu 0.5.25 | 492 t/s | - | 12066 t/s | 0.29 GB |
| Apple M5 32GB | MLX 0.32.1 | 292 t/s | - | 6349 t/s | 1.63 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 358 t/s | - | 4993 t/s | 0.30 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 828 t/s | - | 21227 t/s | 0.29 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 510 t/s | - | 7507 t/s | 1.38 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 476 t/s | - | 15034 t/s | 0.37 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 1116 t/s | - | 35674 t/s | 0.29 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 796 t/s | - | 9050 t/s | 1.50 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 712 t/s | - | 21498 t/s | 0.36 GB |

### LFM 2.5 350M Mirai-L 8-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-350m-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-350M-L
- Model identifier in the uzu SDK: `trymirai/LFM2.5-350M-L`
- Registry id: `liquidai:lfm2.5:350m:mirai:mirai-l:8`
- Mirai-L 8-bit, 350M parameters, 373 MB, 128000 context
- Tool calls: no; reasoning: no


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-350M-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 64. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-350M/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-350M-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-350M-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-350M-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-350M-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 137 t/s | - | 2380 t/s | 0.48 GB |
| Apple M1 16GB | MLX 0.32.1 | 100 t/s | - | 1940 t/s | 1.50 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 100 t/s | - | 2656 t/s | 0.44 GB |
| Apple M2 24GB | uzu 0.5.25 | 196 t/s | - | 3372 t/s | 0.48 GB |
| Apple M2 24GB | MLX 0.32.1 | 148 t/s | - | 2840 t/s | 1.50 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 161 t/s | - | 3921 t/s | 0.44 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 349 t/s | - | 6231 t/s | 0.48 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 283 t/s | - | 4084 t/s | 1.81 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 260 t/s | - | 7158 t/s | 0.44 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 570 t/s | - | 15563 t/s | 0.48 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 461 t/s | - | 6852 t/s | 1.78 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 375 t/s | - | 14516 t/s | 0.44 GB |
| Apple M4 32GB | uzu 0.5.25 | 245 t/s | - | 5129 t/s | 0.48 GB |
| Apple M4 32GB | MLX 0.32.1 | 176 t/s | - | 3723 t/s | 1.50 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 203 t/s | - | 4752 t/s | 0.42 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 479 t/s | - | 10026 t/s | 0.48 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 353 t/s | - | 5798 t/s | 1.78 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 359 t/s | - | 9523 t/s | 0.44 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 709 t/s | - | 18811 t/s | 0.48 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 552 t/s | - | 8098 t/s | 1.78 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 500 t/s | - | 16662 t/s | 0.44 GB |
| Apple M5 32GB | uzu 0.5.25 | 294 t/s | - | 13847 t/s | 0.46 GB |
| Apple M5 32GB | MLX 0.32.1 | 209 t/s | - | 6407 t/s | 1.50 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 257 t/s | - | 4948 t/s | 0.44 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 557 t/s | - | 23390 t/s | 0.46 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 307 t/s | - | 7890 t/s | 1.76 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 327 t/s | - | 18311 t/s | 0.52 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 829 t/s | - | 40043 t/s | 0.46 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 657 t/s | - | 10761 t/s | 1.76 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 614 t/s | - | 28791 t/s | 0.51 GB |

### LFM 2.5 1.2B Instruct Mirai-M 4-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-1-2b-instruct-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-1.2B-Instruct-M
- Model identifier in the uzu SDK: `trymirai/LFM2.5-1.2B-Instruct-M`
- Registry id: `liquidai:lfm2.5:1.2b:instruct:mirai:mirai-m:4`
- Mirai-M 4-bit, 1.2B parameters, 639 MB, 128000 context
- Tool calls: no; reasoning: no


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-1.2B-Instruct-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 64. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-1.2B-Instruct-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-1.2B-Instruct-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-1.2B-Instruct-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-1.2B-Instruct-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 85 t/s | - | 584 t/s | 0.76 GB |
| Apple M1 16GB | MLX 0.32.1 | 73 t/s | - | 623 t/s | 1.24 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 63 t/s | - | 755 t/s | 0.74 GB |
| Apple M2 24GB | uzu 0.5.25 | 121 t/s | - | 831 t/s | 0.76 GB |
| Apple M2 24GB | MLX 0.32.1 | 105 t/s | - | 876 t/s | 1.24 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 96 t/s | - | 1102 t/s | 0.74 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 220 t/s | - | 1561 t/s | 0.76 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 189 t/s | - | 1529 t/s | 1.48 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 164 t/s | - | 2048 t/s | 0.74 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 388 t/s | - | 5066 t/s | 0.76 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 343 t/s | - | 3755 t/s | 1.48 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 267 t/s | - | 4792 t/s | 0.74 GB |
| Apple M4 32GB | uzu 0.5.25 | 146 t/s | - | 1558 t/s | 0.76 GB |
| Apple M4 32GB | MLX 0.32.1 | 131 t/s | - | 1466 t/s | 1.24 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 117 t/s | - | 1443 t/s | 0.75 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 284 t/s | - | 3073 t/s | 0.76 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 265 t/s | - | 2571 t/s | 1.48 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 221 t/s | - | 2801 t/s | 0.74 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 476 t/s | - | 5966 t/s | 0.76 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 429 t/s | - | 4464 t/s | 1.48 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 343 t/s | - | 5509 t/s | 0.74 GB |
| Apple M5 32GB | uzu 0.5.25 | 177 t/s | - | 4423 t/s | 0.74 GB |
| Apple M5 32GB | MLX 0.32.1 | 151 t/s | - | 4131 t/s | 1.24 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 140 t/s | - | 1524 t/s | 0.75 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 357 t/s | - | 8330 t/s | 0.74 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 316 t/s | - | 7173 t/s | 1.48 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 248 t/s | - | 7661 t/s | 0.82 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 583 t/s | - | 14231 t/s | 0.74 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 481 t/s | - | 9272 t/s | 1.48 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 456 t/s | - | 12632 t/s | 0.81 GB |

### LFM 2.5 1.2B Thinking Mirai-M 4-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-1-2b-thinking-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-1.2B-Thinking-M
- Model identifier in the uzu SDK: `trymirai/LFM2.5-1.2B-Thinking-M`
- Registry id: `liquidai:lfm2.5:1.2b:thinking:mirai:mirai-m:4`
- Mirai-M 4-bit, 1.2B parameters, 639 MB, 128000 context
- Tool calls: no; reasoning: yes


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-1.2B-Thinking-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 64. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-1.2B-Thinking](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-1.2B-Thinking-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-1.2B-Thinking-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-1.2B-Thinking-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-1.2B-Thinking-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 84 t/s | - | 585 t/s | 0.76 GB |
| Apple M1 16GB | MLX 0.32.1 | 72 t/s | - | 621 t/s | 1.24 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 61 t/s | - | 746 t/s | 0.74 GB |
| Apple M2 24GB | uzu 0.5.25 | 120 t/s | - | 831 t/s | 0.76 GB |
| Apple M2 24GB | MLX 0.32.1 | 105 t/s | - | 876 t/s | 1.24 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 96 t/s | - | 1106 t/s | 0.74 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 218 t/s | - | 1561 t/s | 0.76 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 186 t/s | - | 1523 t/s | 1.48 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 162 t/s | - | 2030 t/s | 0.74 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 387 t/s | - | 5058 t/s | 0.76 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 338 t/s | - | 3829 t/s | 1.48 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 263 t/s | - | 4792 t/s | 0.74 GB |
| Apple M4 32GB | uzu 0.5.25 | 145 t/s | - | 1557 t/s | 0.76 GB |
| Apple M4 32GB | MLX 0.32.1 | 129 t/s | - | 1498 t/s | 1.24 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 115 t/s | - | 1442 t/s | 0.74 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 283 t/s | - | 3074 t/s | 0.76 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 261 t/s | - | 2666 t/s | 1.48 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 218 t/s | - | 2885 t/s | 0.74 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 475 t/s | - | 5970 t/s | 0.76 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 419 t/s | - | 4497 t/s | 1.48 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 337 t/s | - | 5507 t/s | 0.74 GB |
| Apple M5 32GB | uzu 0.5.25 | 177 t/s | - | 4480 t/s | 0.74 GB |
| Apple M5 32GB | MLX 0.32.1 | 152 t/s | - | 4130 t/s | 1.24 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 144 t/s | - | 1522 t/s | 0.74 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 356 t/s | - | 8494 t/s | 0.74 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 227 t/s | - | 5730 t/s | 1.48 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 208 t/s | - | 5793 t/s | 0.81 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 578 t/s | - | 14120 t/s | 0.74 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 507 t/s | - | 10012 t/s | 1.48 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 447 t/s | - | 13069 t/s | 0.81 GB |

### LFM 2.5 1.2B Instruct Mirai-L 8-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-1-2b-instruct-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-1.2B-Instruct-L
- Model identifier in the uzu SDK: `trymirai/LFM2.5-1.2B-Instruct-L`
- Registry id: `liquidai:lfm2.5:1.2b:instruct:mirai:mirai-l:8`
- Mirai-L 8-bit, 1.2B parameters, 1.2 GB, 128000 context
- Tool calls: no; reasoning: no


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-1.2B-Instruct-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 64. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-1.2B-Instruct-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-1.2B-Instruct-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-1.2B-Instruct-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-1.2B-Instruct-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 47 t/s | - | 731 t/s | 1.28 GB |
| Apple M1 16GB | MLX 0.32.1 | 44 t/s | - | 623 t/s | 1.72 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 38 t/s | - | 820 t/s | 1.37 GB |
| Apple M2 24GB | uzu 0.5.25 | 70 t/s | - | 1035 t/s | 1.28 GB |
| Apple M2 24GB | MLX 0.32.1 | 64 t/s | - | 883 t/s | 1.72 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 59 t/s | - | 1225 t/s | 1.37 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 130 t/s | - | 1951 t/s | 1.28 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 121 t/s | - | 1530 t/s | 1.95 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 110 t/s | - | 2305 t/s | 1.37 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 254 t/s | - | 5072 t/s | 1.28 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 227 t/s | - | 3740 t/s | 1.95 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 187 t/s | - | 4943 t/s | 1.37 GB |
| Apple M4 32GB | uzu 0.5.25 | 81 t/s | - | 1564 t/s | 1.28 GB |
| Apple M4 32GB | MLX 0.32.1 | 75 t/s | - | 1462 t/s | 1.72 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 68 t/s | - | 1488 t/s | 1.37 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 181 t/s | - | 3088 t/s | 1.28 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 165 t/s | - | 2644 t/s | 1.95 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 144 t/s | - | 2960 t/s | 1.37 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 321 t/s | - | 6000 t/s | 1.28 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 285 t/s | - | 4397 t/s | 1.95 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 237 t/s | - | 5684 t/s | 1.37 GB |
| Apple M5 32GB | uzu 0.5.25 | 96 t/s | - | 4989 t/s | 1.31 GB |
| Apple M5 32GB | MLX 0.32.1 | 86 t/s | - | 3826 t/s | 1.72 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 81 t/s | - | 1550 t/s | 1.37 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 214 t/s | - | 9422 t/s | 1.31 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 194 t/s | - | 7033 t/s | 1.95 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 175 t/s | - | 7857 t/s | 1.44 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 377 t/s | - | 17108 t/s | 1.31 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 188 t/s | - | 7531 t/s | 1.95 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 283 t/s | - | 13428 t/s | 1.44 GB |

### LFM 2.5 1.2B Thinking Mirai-L 8-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-1-2b-thinking-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-1.2B-Thinking-L
- Model identifier in the uzu SDK: `trymirai/LFM2.5-1.2B-Thinking-L`
- Registry id: `liquidai:lfm2.5:1.2b:thinking:mirai:mirai-l:8`
- Mirai-L 8-bit, 1.2B parameters, 1.2 GB, 128000 context
- Tool calls: no; reasoning: yes


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-1.2B-Thinking-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 64. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-1.2B-Thinking](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-1.2B-Thinking-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-1.2B-Thinking-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-1.2B-Thinking-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-1.2B-Thinking-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 47 t/s | - | 731 t/s | 1.28 GB |
| Apple M1 16GB | MLX 0.32.1 | 43 t/s | - | 619 t/s | 1.72 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 37 t/s | - | 806 t/s | 1.37 GB |
| Apple M2 24GB | uzu 0.5.25 | 70 t/s | - | 1035 t/s | 1.28 GB |
| Apple M2 24GB | MLX 0.32.1 | 64 t/s | - | 882 t/s | 1.72 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 59 t/s | - | 1234 t/s | 1.37 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 130 t/s | - | 1952 t/s | 1.28 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 120 t/s | - | 1540 t/s | 1.95 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 108 t/s | - | 2298 t/s | 1.37 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 253 t/s | - | 5073 t/s | 1.28 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 227 t/s | - | 3799 t/s | 1.95 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 185 t/s | - | 4924 t/s | 1.37 GB |
| Apple M4 32GB | uzu 0.5.25 | 81 t/s | - | 1565 t/s | 1.28 GB |
| Apple M4 32GB | MLX 0.32.1 | 75 t/s | - | 1482 t/s | 1.72 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 67 t/s | - | 1490 t/s | 1.37 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 181 t/s | - | 3088 t/s | 1.28 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 165 t/s | - | 2668 t/s | 1.95 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 143 t/s | - | 2930 t/s | 1.37 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 320 t/s | - | 5999 t/s | 1.28 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 284 t/s | - | 4481 t/s | 1.95 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 235 t/s | - | 5707 t/s | 1.37 GB |
| Apple M5 32GB | uzu 0.5.25 | 98 t/s | - | 5049 t/s | 1.31 GB |
| Apple M5 32GB | MLX 0.32.1 | 88 t/s | - | 3848 t/s | 1.72 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 83 t/s | - | 1563 t/s | 1.37 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 214 t/s | - | 9476 t/s | 1.31 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 141 t/s | - | 5486 t/s | 1.95 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 126 t/s | - | 6031 t/s | 1.43 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 372 t/s | - | 16871 t/s | 1.31 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 331 t/s | - | 9526 t/s | 1.95 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 302 t/s | - | 13354 t/s | 1.43 GB |

### LFM 2.5 2.6B Mirai-M 4-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-2-6b-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-2.6B-M
- Model identifier in the uzu SDK: `trymirai/LFM2.5-2.6B-M`
- Registry id: `liquidai:lfm2.5:2.6b:mirai:mirai-m:4`
- Mirai-M 4-bit, 2.6B parameters, 1.5 GB, 131072 context
- Tool calls: no; reasoning: no


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-2.6B-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 64. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-2.6B-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-2.6B-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-2.6B-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-2.6B-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 38 t/s | - | 250 t/s | 1.55 GB |
| Apple M1 16GB | MLX 0.32.1 | 33 t/s | - | 263 t/s | 2.18 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 29 t/s | - | 317 t/s | 1.69 GB |
| Apple M2 24GB | uzu 0.5.25 | 55 t/s | - | 355 t/s | 1.55 GB |
| Apple M2 24GB | MLX 0.32.1 | 49 t/s | - | 370 t/s | 2.18 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 45 t/s | - | 471 t/s | 1.69 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 102 t/s | - | 670 t/s | 1.55 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 89 t/s | - | 675 t/s | 2.39 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 76 t/s | - | 877 t/s | 1.69 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 185 t/s | - | 2208 t/s | 1.55 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 171 t/s | - | 1817 t/s | 2.39 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 132 t/s | - | 2055 t/s | 1.69 GB |
| Apple M4 32GB | uzu 0.5.25 | 65 t/s | - | 665 t/s | 1.55 GB |
| Apple M4 32GB | MLX 0.32.1 | 59 t/s | - | 636 t/s | 2.18 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 52 t/s | - | 613 t/s | 1.69 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 130 t/s | - | 1315 t/s | 1.55 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 125 t/s | - | 1198 t/s | 2.39 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 104 t/s | - | 1218 t/s | 1.69 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 230 t/s | - | 2563 t/s | 1.55 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 214 t/s | - | 2147 t/s | 2.39 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 167 t/s | - | 2389 t/s | 1.69 GB |
| Apple M5 32GB | uzu 0.5.25 | 80 t/s | - | 2002 t/s | 1.54 GB |
| Apple M5 32GB | MLX 0.32.1 | 68 t/s | - | 2003 t/s | 2.18 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 63 t/s | - | 657 t/s | 1.68 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 164 t/s | - | 3879 t/s | 1.54 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 108 t/s | - | 2779 t/s | 2.39 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 98 t/s | - | 2506 t/s | 1.76 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 282 t/s | - | 7030 t/s | 1.54 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 252 t/s | - | 5481 t/s | 2.39 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 222 t/s | - | 5431 t/s | 1.78 GB |

### LFM 2.5 2.6B Mirai-L 8-bit

- Page: https://trymirai.com/local-models/liquidai-lfm2-5-2-6b-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/LFM2.5-2.6B-L
- Model identifier in the uzu SDK: `trymirai/LFM2.5-2.6B-L`
- Registry id: `liquidai:lfm2.5:2.6b:mirai:mirai-l:8`
- Mirai-L 8-bit, 2.6B parameters, 2.8 GB, 131072 context
- Tool calls: no; reasoning: no


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/LFM2.5-2.6B-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 64. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.



#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [LiquidAI/LFM2.5-2.6B](https://huggingface.co/LiquidAI/LFM2.5-2.6B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/LFM2.5-2.6B-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/LFM2.5-2.6B-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/LFM2.5-2.6B-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/LFM2.5-2.6B-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M1 16GB | uzu 0.5.25 | 21 t/s | - | 312 t/s | 2.77 GB |
| Apple M1 16GB | MLX 0.32.1 | 20 t/s | - | 262 t/s | 3.36 GB |
| Apple M1 16GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 19 t/s | - | 345 t/s | 2.80 GB |
| Apple M2 24GB | uzu 0.5.25 | 31 t/s | - | 441 t/s | 2.77 GB |
| Apple M2 24GB | MLX 0.32.1 | 30 t/s | - | 374 t/s | 3.36 GB |
| Apple M2 24GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 30 t/s | - | 527 t/s | 2.80 GB |
| Apple M2 Pro 32GB | uzu 0.5.25 | 59 t/s | - | 836 t/s | 2.77 GB |
| Apple M2 Pro 32GB | MLX 0.32.1 | 56 t/s | - | 679 t/s | 3.37 GB |
| Apple M2 Pro 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 55 t/s | - | 998 t/s | 2.80 GB |
| Apple M3 Max 48GB | uzu 0.5.25 | 116 t/s | - | 2224 t/s | 2.77 GB |
| Apple M3 Max 48GB | MLX 0.32.1 | 109 t/s | - | 1785 t/s | 3.37 GB |
| Apple M3 Max 48GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 97 t/s | - | 2168 t/s | 2.80 GB |
| Apple M4 32GB | uzu 0.5.25 | 36 t/s | - | 668 t/s | 2.77 GB |
| Apple M4 32GB | MLX 0.32.1 | 34 t/s | - | 627 t/s | 3.36 GB |
| Apple M4 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 33 t/s | - | 635 t/s | 2.80 GB |
| Apple M4 Pro 64GB | uzu 0.5.25 | 81 t/s | - | 1321 t/s | 2.77 GB |
| Apple M4 Pro 64GB | MLX 0.32.1 | 77 t/s | - | 1180 t/s | 3.37 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 73 t/s | - | 1273 t/s | 2.80 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 147 t/s | - | 2575 t/s | 2.77 GB |
| Apple M4 Max 128GB | MLX 0.32.1 | 138 t/s | - | 2106 t/s | 3.37 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 123 t/s | - | 2479 t/s | 2.80 GB |
| Apple M5 32GB | uzu 0.5.25 | 43 t/s | - | 2228 t/s | 2.79 GB |
| Apple M5 32GB | MLX 0.32.1 | 41 t/s | - | 1830 t/s | 3.36 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 41 t/s | - | 675 t/s | 2.77 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 92 t/s | - | 4370 t/s | 2.79 GB |
| Apple M5 Pro 64GB | MLX 0.32.1 | 65 t/s | - | 2603 t/s | 3.37 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 66 t/s | - | 2750 t/s | 2.86 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 171 t/s | - | 8275 t/s | 2.79 GB |
| Apple M5 Max 128GB | MLX 0.32.1 | 155 t/s | - | 5285 t/s | 3.37 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 155 t/s | - | 6292 t/s | 2.90 GB |

## Muse-Glimmer

### Muse-Glimmer 30B Mirai-M 4-bit

- Page: https://trymirai.com/local-models/meta-muse-glimmer-30b-mirai-mirai-m-4
- Hugging Face repo: https://huggingface.co/trymirai/Muse-Glimmer-30B-M
- Model identifier in the uzu SDK: `trymirai/Muse-Glimmer-30B-M`
- Registry id: `meta:muse-glimmer:30b:mirai:mirai-m:4`
- Mirai-M 4-bit, 30B parameters, 16.5 GB, 131072 context
- Tool calls: yes; reasoning: yes


Mirai-M is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Muse-Glimmer-30B-M
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Medium uses 4-bit asymmetric integer quantization with 4-bit zero points, bfloat16 scales, and group size 64. Block-diagonal Random Hadamard Transforms are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization followed by quantization-aware distillation.

On our internal KL evals, this checkpoint is comparable to Unsloth's [`UD-Q4_K_XL`](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF/blob/main/Muse-Glimmer-30B-UD-Q4_K_XL.gguf) GGUF.

#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [meta-models/Muse-Glimmer-30B](https://huggingface.co/meta-models/Muse-Glimmer-30B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Muse-Glimmer-30B-M") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Muse-Glimmer-30B-M');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Muse-Glimmer-30B-M")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Muse-Glimmer-30B-M".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M4 Pro 64GB | uzu 0.5.25 | 15 t/s | - | 127 t/s | 14.92 GB |
| Apple M4 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 9 t/s | - | 116 t/s | 12.79 GB |
| Apple M4 Max 128GB | uzu 0.5.25 | 28 t/s | - | 250 t/s | 14.92 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 17 t/s | - | 230 t/s | 12.86 GB |
| Apple M5 32GB | uzu 0.5.25 | 8 t/s | - | 208 t/s | 14.90 GB |
| Apple M5 32GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 8 t/s | - | 60 t/s | 12.82 GB |
| Apple M5 Pro 64GB | uzu 0.5.25 | 18 t/s | - | 422 t/s | 14.90 GB |
| Apple M5 Pro 64GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 20 t/s | - | 353 t/s | 12.90 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 33 t/s | - | 803 t/s | 14.90 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 33 t/s | - | 683 t/s | 12.87 GB |

### Muse-Glimmer 30B Mirai-L 8-bit

- Page: https://trymirai.com/local-models/meta-muse-glimmer-30b-mirai-mirai-l-8
- Hugging Face repo: https://huggingface.co/trymirai/Muse-Glimmer-30B-L
- Model identifier in the uzu SDK: `trymirai/Muse-Glimmer-30B-L`
- Registry id: `meta:muse-glimmer:30b:mirai:mirai-l:8`
- Mirai-L 8-bit, 30B parameters, 30.2 GB, 131072 context
- Tool calls: yes; reasoning: yes


Mirai-L is on the size-KL Pareto frontier: we found no checkpoint that is smaller while also having lower KL divergence.

*Evaluation data mixture: 45% public agentic, 30% public SFT/long-context, 25% private chat data.*

#### Quickstart

If you are on macOS, the easiest way is to install the `mirai` Homebrew package and then run the CLI:

```sh
brew install mirai
mirai --model trymirai/Muse-Glimmer-30B-L
```

Currently only Apple silicon inference is supported. If you want to build things from source, read this [overview](https://github.com/trymirai/uzu/blob/how-to/docs/how-to-run-uzu.md).


#### Method

Mirai Large uses 8-bit symmetric integer quantization with bfloat16 scales and group size 64. Block-diagonal Random Hadamard Transforms with block size 32 are used to reduce activation and weight outliers. The checkpoint was prepared with post-training quantization.

On our internal KL evals, this checkpoint is comparable to Unsloth's [`UD-Q8_K_XL`](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF/blob/main/Muse-Glimmer-30B-UD-Q8_K_XL.gguf) GGUF.

#### Citation

If you find our work helpful, feel free to give us a cite.

```bibtex
@misc{mirai-quant,
    title  = {{Mirai Quantization}: Redefining the speed-quality frontier for local LLMs on Apple silicon},
    author = {Artur Chakhvadze and Ryan Mathieu and Roman Knyazhitskiy and Nikolai Voinilenko and Chen-Chen Yeh and Artur Mullakhmetov and Eugene Bokhan and others},
    note   = {In collaboration with others at Mirai Labs},
    month  = {June},
    year   = {2026},
    url    = {https://trymirai.com/blog/quantization}
}
```

#### Original model

This is a quantized version of [meta-models/Muse-Glimmer-30B](https://huggingface.co/meta-models/Muse-Glimmer-30B). For architecture details, intended use, evaluations, and limitations, see the [original model card](https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/README.md).

Install and run with uzu:

Swift:

```
https://github.com/trymirai/uzu
```

```
import Foundation
import Uzu

public func runChat() async throws {
    let engineConfig = EngineConfig.create()
    let engine = try await Engine.create(config: engineConfig)
    
    guard let model = try await engine.model(identifier: "trymirai/Muse-Glimmer-30B-L") else {
        return
    }
    for try await update in try await engine.download(model: model).iterator() {
        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
        fflush(stdout)
    }
    print()
    
    let messages = [
        ChatMessage.system().withText(text: "You are a helpful assistant"),
        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
    ]
    let session = try await engine.chat(model: model, config: .create())
    let stream = await session.replyWithStream(input: messages, config: .create())
    var message: ChatMessage? = nil
    for try await update in stream.iterator() {
        switch update {
        case .replies(let replies):
            let reply = replies.last
            message = reply?.message
            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
        case .error(let error):
            print("Error: \(error)")
        }
    }
    print("Reasoning: \(message?.reasoning() ?? "empty")")
    print("Text: \(message?.text() ?? "empty")")
}

```

TypeScript:

```
pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu
```

```
import {
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunkError,
    ChatSessionStreamChunkReplies,
    Engine,
    EngineConfig
} from '@trymirai/uzu';

async function main() {
    let engineConfig = EngineConfig.create();
    let engine = await Engine.create(engineConfig);

    let model = await engine.model('trymirai/Muse-Glimmer-30B-L');
    if (!model) {
        throw new Error('Model not found');
    }
    for await (const update of await engine.download(model)) {
        process.stdout.write(`\rDownload progress: ${(update.progress * 100).toFixed(2)}%`);
    }
    console.log();

    let messages = [
        ChatMessage.system().withText('You are a helpful assistant'),
        ChatMessage.user().withText('Tell me a short, funny story about a robot')
    ];
    let session = await engine.chat(model, ChatConfig.create());
    let stream = await session.replyWithStream(messages, ChatReplyConfig.create());
    let message: ChatMessage | undefined;
    for await (const chunk of stream) {
        if (chunk instanceof ChatSessionStreamChunkReplies) {
            message = chunk.replies[0]?.message;
            console.log('Generated tokens: ', chunk.replies[0]?.stats.tokensCountOutput);
        } else if (chunk instanceof ChatSessionStreamChunkError) {
            console.error('Error: ', chunk.error);
        }
    }
    console.log('Reasoning: ', message?.reasoning);
    console.log('Text: ', message?.text);
}

main().catch((error) => {
    console.error(error);
});

```

Python:

```
uv add uzu
```

```
import asyncio

from uzu import (
    ChatConfig,
    ChatMessage,
    ChatReplyConfig,
    ChatSessionStreamChunk,
    Engine,
    EngineConfig,
)


async def main() -> None:
    engine_config = EngineConfig.create()
    engine = await Engine.create(engine_config)

    model = await engine.model("trymirai/Muse-Glimmer-30B-L")
    if model is None:
        raise RuntimeError("Model not found")
    async for update in (await engine.download(model)).iterator():
        print(f"\rDownload progress: {update.progress:.2%}", end="", flush=True)
    print()

    messages = [
        ChatMessage.system().with_text("You are a helpful assistant"),
        ChatMessage.user().with_text("Tell me a short, funny story about a robot"),
    ]
    session = await engine.chat(model, ChatConfig.create())
    stream = await session.reply_with_stream(messages, ChatReplyConfig.create())
    message: ChatMessage | None = None
    async for chunk in stream.iterator():
        if isinstance(chunk, ChatSessionStreamChunk.Replies):
            replies = chunk.replies
            if replies:
                reply = replies[0]
                message = reply.message
                print(f"Generated tokens: {reply.stats.tokens_count_output}")
        elif isinstance(chunk, ChatSessionStreamChunk.Error):
            print(f"Error: {chunk.error}")
    if message is not None:
        print(f"Reasoning: {message.reasoning}")
        print(f"Text: {message.text}")


if __name__ == "__main__":
    asyncio.run(main())

```

Rust:

```
cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full
```

```
use std::io::{self, Write};

use uzu::{
    engine::{Engine, EngineConfig},
    session::chat::ChatSessionStreamChunk,
    types::session::chat::{ChatConfig, ChatMessage, ChatReplyConfig},
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let engine_config = EngineConfig::default();
    let engine = Engine::new(engine_config).await?;

    let model = engine.model("trymirai/Muse-Glimmer-30B-L".to_string()).await?.ok_or("Model not found")?;
    let downloader = engine.download(&model).await?;
    while let Some(update) = downloader.next().await {
        print!("\r\u{001B}[2KDownload progress: {:.2}%", update.progress() * 100.0);
        io::stdout().flush()?;
    }
    println!();

    let messages = vec![
        ChatMessage::system().with_text("You are a helpful assistant".to_string()),
        ChatMessage::user().with_text("Tell me a short, funny story about a robot".to_string()),
    ];
    let session = engine.chat(model, ChatConfig::default()).await?;
    let stream = session.reply_with_stream(messages, ChatReplyConfig::default()).await;
    let mut last_message: Option<ChatMessage> = None;
    while let Some(chunk) = stream.next().await {
        match chunk {
            ChatSessionStreamChunk::Replies {
                replies,
            } => {
                if let Some(reply) = replies.first() {
                    last_message = Some(reply.message.clone());
                    println!("Generated tokens: {}", reply.stats.tokens_count_output.unwrap_or_default());
                }
            },
            ChatSessionStreamChunk::Error {
                error,
            } => {
                println!("Error: {error}");
            },
        }
    }
    if let Some(message) = last_message {
        println!("Reasoning: {}", message.reasoning().unwrap_or_default());
        println!("Text: {}", message.text().unwrap_or_default());
    }

    Ok(())
}

```

Benchmarks (benchmarked 3 Sept 2026). Output and input are tokens per second on a fixed prompt; speculative output is uzu with speculative decoding where a draft model was available:

| Device | Engine | Output | Output (speculative) | Input | Resident memory |
|---|---|---|---|---|---|
| Apple M4 Max 128GB | uzu 0.5.25 | 16 t/s | - | 249 t/s | 27.75 GB |
| Apple M4 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 15 t/s | - | 255 t/s | 30.50 GB |
| Apple M5 Max 128GB | uzu 0.5.25 | 19 t/s | - | 915 t/s | 27.87 GB |
| Apple M5 Max 128GB | llama.cpp 0.3.0-dev+c1d0e7a00 | 16 t/s | - | 638 t/s | 24.44 GB |

## Product

- [Local LLM library for Apple Silicon](https://trymirai.com/local-models): 24 Qwen, LFM and Muse-Glimmer checkpoints in 4-bit (Mirai-M) and 8-bit (Mirai-L) for Apple Silicon, with measured uzu tok/s for every checkpoint.
- [uzu: On-device LLM inference engine for iPhone, iPad & Mac](https://trymirai.com/inference-runtime): uzu runs LLMs on iPhone, iPad and Mac faster than MLX and llama.cpp. Benchmarks published for each release.
- [Conversion and optimization toolkit](https://trymirai.com/conversion-optimization-toolkit): Convert and optimize your model for iPhone, iPad and Mac. One command to get your model running on 2 billion Apple devices.
- [Mirai for macOS: Faster alternative to Ollama & LM Studio](https://trymirai.com/chat-for-mac): Chat with your favorite AI models directly on your Mac. Privately and securely. Built natively for macOS and Apple Silicon.

## Research

- [Speculative decoding in uzu](https://trymirai.com/blog/speculative-decoding-in-uzu): We are releasing our speculative decoding implementation in Uzu. Initially for Qwen3.6 27B, with support for Qwen3.8 27B and Muse Glimmer coming soon. On Apple M5-series chips, we outperform MTPLX by almost 2x, and llama.cpp by over 3x at comparable quantization levels, with the strongest gains acheived on mathematical reasoning and coding tasks.
- [Trees from Marginals: Autoregressive drafting with factorized priors](https://arxiv.org/abs/2607.06763): Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter.
- [Sparse Buffers for KV Cache](https://trymirai.com/blog/sparse-buffers-for-kv-cache): How uzu backs the KV cache with Metal sparse buffers so memory grows page by page instead of committing 4.5 GiB up front for a 32k-context Qwen3-4B.
- [Introducing Mirai Quantization: Redefining the speed-quality frontier for local LLMs on Apple silicon.](https://trymirai.com/blog/quantization): Mirai-M (4-bit) and Mirai-L (8-bit): a co-designed quantization and inference stack with 40-60% more tokens per second than llama.cpp and MLX at the same quality level.

## Company

- [On-Device AI for Apple Silicon: uzu & Models | Mirai Labs](https://trymirai.com/): Mirai Labs builds uzu, a Rust LLM engine for iPhone, iPad and Mac, and 4-bit/8-bit Qwen, LFM and Muse-Glimmer models benchmarked vs MLX and llama.cpp.
- [About Mirai Labs: the On-Device AI Lab for Apple Silicon](https://trymirai.com/about-us): Mirai Labs: 16-person on-device AI lab building uzu and Mirai quantization for Apple Silicon. Founded by Dima Shvets and Alexey Moiseenkov, Uncork-backed.
- [Careers](https://trymirai.com/careers): Join a small, senior team building the full on-device stack to achieve realtime local intelligence.
- [Contact Mirai Labs: uzu, models, Mac app & SDK](https://trymirai.com/i-am-interested): Contact Mirai Labs. Email contact@trymirai.com or use the form to tell us what interests you in Mirai.
- [Machine Learning Engineer](https://trymirai.com/careers/machine-learning-engineer-model-optimization): Apply for the machine learning engineer role for models optimization at Mirai. Join a small, senior team building the full on-device stack to achieve realtime local intelligence.
- [Machine Learning Engineer](https://trymirai.com/careers/machine-learning-engineer): Apply for the machine learning engineer role at Mirai. Join a small, senior team building the full on-device stack to achieve realtime local intelligence.
- [Inference engineer](https://trymirai.com/careers/inference-engineer): Apply for the inference engineer role at Mirai. Join a small, senior team building the full on-device stack to achieve realtime local intelligence.

## Links

- [Docs](https://docs.trymirai.com/)
- [uzu on GitHub](https://github.com/trymirai/uzu)
- [lalamo on GitHub](https://github.com/trymirai/lalamo)
- [Platform](https://platform.trymirai.com/)
- [X (Twitter)](https://x.com/trymirai)
- [GitHub](https://github.com/trymirai)
- [Hugging Face](https://huggingface.co/trymirai)
- [LinkedIn](https://www.linkedin.com/company/trymirai)
- [Discord](https://discord.com/invite/trymirai)