Qwen 3

Name: Qwen/Qwen3-8B-MLX-8bit
Author: Alibaba

Qwen/Qwen3-8B-MLX-8bit

Vendor: Alibaba
Quantization: MLX 8-bit
Parameters: 8B
Size: 7.9 GB

$ brew install mirai$ mirai --model Qwen/Qwen3-8B-MLX-8bit

Benchmarks

1.43 s

lower is better ↓

1.43 s

uzu

1.55 s

mlx

1.49 s

llamacpp

55 t/s

higher is better ↑

55 t/s

uzu

57 t/s

mlx

42 t/s

llamacpp

8.81 GB

lower is better ↓

8.81 GB

18.99 GB

10.43 GB

0.31.2 llama.cpp0.3.23/1,312 input tokens/512 output tokens

Benchmarked 7 Aug 2026

Integrate with SDK

Choose framework

Run the following command to install Mirai SDK

https://github.com/trymirai/uzu-swift

Apply code

1import Foundation2import Uzu34public func runChat() async throws {5    let engineConfig = EngineConfig.create()6    let engine = try await Engine.create(config: engineConfig)7    8    guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {9        return10    }11    for try await update in try await engine.download(model: model).iterator() {12        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")13        fflush(stdout)14    }15    print()16    17    let messages = [18        ChatMessage.system().withText(text: "You are a helpful assistant"),19        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")20    ]21    let session = try await engine.chat(model: model, config: .create())22    let stream = await session.replyWithStream(input: messages, config: .create())23    var message: ChatMessage? = nil24    for try await update in stream.iterator() {25        switch update {26        case .replies(let replies):27            let reply = replies.last28            message = reply?.message29            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")30        case .error(let error):31            print("Error: \(error)")32        }33    }34    print("Reasoning: \(message?.reasoning() ?? "empty")")35    print("Text: \(message?.text() ?? "empty")")36}37

1Choose framework

2Run the following command to install Mirai SDK

https://github.com/trymirai/uzu-swift

3Apply code

1	import Foundation
2	import Uzu
3
4	public func runChat() async throws {
5	let engineConfig = EngineConfig.create()
6	let engine = try await Engine.create(config: engineConfig)
7
8	guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {
9	return
10	}
11	for try await update in try await engine.download(model: model).iterator() {
12	print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
13	fflush(stdout)
14	}
15	print()
16
17	let messages = [
18	ChatMessage.system().withText(text: "You are a helpful assistant"),
19	ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
20	]
21	let session = try await engine.chat(model: model, config: .create())
22	let stream = await session.replyWithStream(input: messages, config: .create())
23	var message: ChatMessage? = nil
24	for try await update in stream.iterator() {
25	switch update {
26	case .replies(let replies):
27	let reply = replies.last
28	message = reply?.message
29	print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
30	case .error(let error):
31	print("Error: \(error)")
32	}
33	}
34	print("Reasoning: \(message?.reasoning() ?? "empty")")
35	print("Text: \(message?.text() ?? "empty")")
36	}
37

Details

An 8-bit quantized version of Qwen3-8B, optimized for Apple Silicon via the MLX framework. Part of the Qwen3 generation of large language models from Alibaba's Qwen team, this variant brings efficient on-device inference to Mac users without sacrificing the core capabilities of the full-precision model.

Architecture & Specs

Qwen3-8B is a causal language model with 8.2 billion parameters (6.95B non-embedding), featuring 36 layers and grouped-query attention (32 Q heads, 8 KV heads). It supports a native context length of 32,768 tokens, extensible to 131,072 tokens using YaRN rope scaling.

Key Capabilities

Dual-mode reasoning is a standout feature of Qwen3. The model supports seamless switching between a *thinking mode* — where it performs step-by-step reasoning wrapped in `<think>...</think>` blocks — and a *non-thinking mode* for fast, general-purpose dialogue. Users can toggle between modes via the `enable_thinking` parameter or inline `/think` and `/no_think` tags within conversations.

Beyond reasoning, Qwen3-8B delivers strong performance across:

Code generation and mathematics, leveraging its deep reasoning capabilities
Creative writing, role-playing, and multi-turn dialogue with improved human preference alignment
Agentic workflows and tool calling, with precise external tool integration in both thinking modes
Multilingual support across 100+ languages and dialects, including translation and multilingual instruction following

Usage

This model is designed for use with the `mlx_lm` Python package (≥ 0.25.2) and is compatible with `transformers` (≥ 4.52.4). It works well for local inference on Apple Silicon Macs, offering a practical balance of quality and efficiency through 8-bit quantization.

Provenance

Released under the Apache 2.0 license by the Qwen team. For benchmarks, detailed training methodology, and deployment guidance, refer to the official Qwen3 blog and documentation.

Explore all local models