Qwen 3

Name: Qwen/Qwen3-8B-MLX-4bit
Author: Alibaba

Qwen/Qwen3-8B-MLX-4bit

Vendor: Alibaba
Quantization: MLX 4-bit
Parameters: 8B
Size: 4.1 GB

$ brew install mirai$ mirai --model Qwen/Qwen3-8B-MLX-4bit

Benchmarks

1.42 s

lower is better ↓

1.42 s

uzu

1.52 s

mlx

1.55 s

llamacpp

94 t/s

higher is better ↑

94 t/s

uzu

96 t/s

mlx

68 t/s

llamacpp

5.00 GB

lower is better ↓

5.00 GB

18.99 GB

4.87 GB

0.31.2 llama.cpp0.3.23/1,312 input tokens/512 output tokens

Benchmarked 7 Aug 2026

Integrate with SDK

Choose framework

Run the following command to install Mirai SDK

https://github.com/trymirai/uzu-swift

Apply code

1import Foundation2import Uzu34public func runChat() async throws {5    let engineConfig = EngineConfig.create()6    let engine = try await Engine.create(config: engineConfig)7    8    guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {9        return10    }11    for try await update in try await engine.download(model: model).iterator() {12        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")13        fflush(stdout)14    }15    print()16    17    let messages = [18        ChatMessage.system().withText(text: "You are a helpful assistant"),19        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")20    ]21    let session = try await engine.chat(model: model, config: .create())22    let stream = await session.replyWithStream(input: messages, config: .create())23    var message: ChatMessage? = nil24    for try await update in stream.iterator() {25        switch update {26        case .replies(let replies):27            let reply = replies.last28            message = reply?.message29            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")30        case .error(let error):31            print("Error: \(error)")32        }33    }34    print("Reasoning: \(message?.reasoning() ?? "empty")")35    print("Text: \(message?.text() ?? "empty")")36}37

1Choose framework

2Run the following command to install Mirai SDK

https://github.com/trymirai/uzu-swift

3Apply code

1	import Foundation
2	import Uzu
3
4	public func runChat() async throws {
5	let engineConfig = EngineConfig.create()
6	let engine = try await Engine.create(config: engineConfig)
7
8	guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {
9	return
10	}
11	for try await update in try await engine.download(model: model).iterator() {
12	print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
13	fflush(stdout)
14	}
15	print()
16
17	let messages = [
18	ChatMessage.system().withText(text: "You are a helpful assistant"),
19	ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
20	]
21	let session = try await engine.chat(model: model, config: .create())
22	let stream = await session.replyWithStream(input: messages, config: .create())
23	var message: ChatMessage? = nil
24	for try await update in stream.iterator() {
25	switch update {
26	case .replies(let replies):
27	let reply = replies.last
28	message = reply?.message
29	print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
30	case .error(let error):
31	print("Error: \(error)")
32	}
33	}
34	print("Reasoning: \(message?.reasoning() ?? "empty")")
35	print("Text: \(message?.text() ?? "empty")")
36	}
37

Details

A 4-bit quantized version of Qwen3-8B, optimized for Apple Silicon via the MLX framework. Part of the Qwen3 generation of large language models from Alibaba's Qwen team, this variant brings powerful reasoning and conversational capabilities to local Mac inference with a reduced memory footprint.

Architecture & Specs

Qwen3-8B is a causal language model with 8.2 billion parameters (6.95B non-embedding). It features 36 transformer layers with grouped-query attention (32 Q heads, 8 KV heads) and supports a native context length of 32,768 tokens — extendable to 131,072 tokens via YaRN rope scaling. This MLX 4-bit quantization makes it practical for on-device use with `mlx_lm`.

Thinking & Non-Thinking Modes

A standout feature of Qwen3 is its ability to seamlessly switch between thinking mode and non-thinking mode within a single model. In thinking mode, the model performs step-by-step reasoning (wrapped in `<think>...</think>` blocks), ideal for math, coding, and complex logic. In non-thinking mode, it behaves more like a traditional instruct model — fast, direct, and efficient for general dialogue. Users can toggle between modes via the `enable_thinking` parameter or even mid-conversation using `/think` and `/no_think` tags.

Key Capabilities

Strong reasoning in math, code generation, and commonsense logic — surpassing QwQ and Qwen2.5 instruct models in their respective domains
Agent and tool-calling support, with precise external tool integration compatible with frameworks like Qwen-Agent and MCP servers
Multilingual proficiency across 100+ languages and dialects, with strong translation and multilingual instruction-following
Creative and conversational quality, including role-playing, multi-turn dialogues, and nuanced instruction following

Intended Use

Best suited for developers and researchers running inference locally on Apple Silicon Macs who need a capable, general-purpose LLM with advanced reasoning. Requires `mlx_lm ≥ 0.25.2` and `transformers ≥ 4.52.4`. Licensed under Apache 2.0.

Explore all local models