Qwen 3

Name: Qwen/Qwen3-4B-MLX-8bit
Author: Alibaba

Qwen/Qwen3-4B-MLX-8bit

Vendor: Alibaba
Quantization: MLX 8-bit
Parameters: 4B
Size: 3.9 GB

$ brew install mirai$ mirai --model Qwen/Qwen3-4B-MLX-8bit

Benchmarks

0.81 s

lower is better ↓

0.81 s

uzu

0.89 s

mlx

0.86 s

llamacpp

92 t/s

higher is better ↑

92 t/s

uzu

93 t/s

mlx

66 t/s

llamacpp

4.79 GB

lower is better ↓

4.79 GB

18.42 GB

5.10 GB

0.31.2 llama.cpp0.3.23/1,312 input tokens/512 output tokens

Benchmarked 7 Aug 2026

Integrate with SDK

Choose framework

Run the following command to install Mirai SDK

https://github.com/trymirai/uzu-swift

Apply code

1import Foundation2import Uzu34public func runChat() async throws {5    let engineConfig = EngineConfig.create()6    let engine = try await Engine.create(config: engineConfig)7    8    guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {9        return10    }11    for try await update in try await engine.download(model: model).iterator() {12        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")13        fflush(stdout)14    }15    print()16    17    let messages = [18        ChatMessage.system().withText(text: "You are a helpful assistant"),19        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")20    ]21    let session = try await engine.chat(model: model, config: .create())22    let stream = await session.replyWithStream(input: messages, config: .create())23    var message: ChatMessage? = nil24    for try await update in stream.iterator() {25        switch update {26        case .replies(let replies):27            let reply = replies.last28            message = reply?.message29            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")30        case .error(let error):31            print("Error: \(error)")32        }33    }34    print("Reasoning: \(message?.reasoning() ?? "empty")")35    print("Text: \(message?.text() ?? "empty")")36}37

1Choose framework

2Run the following command to install Mirai SDK

https://github.com/trymirai/uzu-swift

3Apply code

1	import Foundation
2	import Uzu
3
4	public func runChat() async throws {
5	let engineConfig = EngineConfig.create()
6	let engine = try await Engine.create(config: engineConfig)
7
8	guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {
9	return
10	}
11	for try await update in try await engine.download(model: model).iterator() {
12	print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")
13	fflush(stdout)
14	}
15	print()
16
17	let messages = [
18	ChatMessage.system().withText(text: "You are a helpful assistant"),
19	ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")
20	]
21	let session = try await engine.chat(model: model, config: .create())
22	let stream = await session.replyWithStream(input: messages, config: .create())
23	var message: ChatMessage? = nil
24	for try await update in stream.iterator() {
25	switch update {
26	case .replies(let replies):
27	let reply = replies.last
28	message = reply?.message
29	print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")
30	case .error(let error):
31	print("Error: \(error)")
32	}
33	}
34	print("Reasoning: \(message?.reasoning() ?? "empty")")
35	print("Text: \(message?.text() ?? "empty")")
36	}
37

Details

An 8-bit quantized version of Qwen3-4B, optimized for efficient inference on Apple Silicon using the MLX framework. Part of the Qwen3 generation of large language models from Alibaba's Qwen team, this variant brings powerful reasoning and conversational capabilities to local Mac hardware.

Key Features

Dual-mode reasoning: Seamlessly switch between a thinking mode — where the model engages in step-by-step reasoning for math, coding, and logic tasks — and a non-thinking mode for fast, general-purpose dialogue. Both modes are available within a single model via the `enable_thinking` toggle or in-prompt `/think` and `/no_think` commands.
4 billion parameters (3.6B non-embedding) across 36 layers, using Grouped Query Attention (32 Q heads, 8 KV heads).
32,768 token native context, extendable to 131,072 tokens using YaRN rotary position embedding scaling.
100+ language support, with strong multilingual instruction following and translation capabilities.
Agent and tool-calling proficiency, with reliable integration into external tools and frameworks like Qwen-Agent and MCP-compatible servers.

Architecture & Quantization

The base Qwen3-4B model is a causal language model that went through both pretraining and post-training stages. This MLX release applies 8-bit quantization, reducing memory footprint while retaining high output quality — ideal for on-device inference via `mlx_lm` (≥ 0.25.2).

Ideal Use Cases

Local AI assistants on macOS with Apple Silicon
Complex reasoning, code generation, and math problem-solving (thinking mode)
Lightweight, responsive chat and creative writing (non-thinking mode)
Multilingual applications and agentic workflows with tool integration

Provenance

Released by the Qwen team under the Apache 2.0 license. Compatible with `transformers` (≥ 4.52.4) and `mlx_lm` for straightforward integration.

Explore all local models