Qwen/Qwen3-8B-MLX-4bit
Run locally on Apple devices with Mirai
- Type
- local
- From
- Alibaba
- Quantization
- MLX 4-bit
- Parameters
- 8B
- Size
- 4.1 GB
- Source
A 4-bit quantized version of Qwen3-8B, optimized for Apple Silicon via the MLX framework. Part of the Qwen3 generation of large language models from Alibaba's Qwen team, this variant brings powerful reasoning and conversational capabilities to local Mac inference with a reduced memory footprint.
Architecture & Specs
Qwen3-8B is a causal language model with 8.2 billion parameters (6.95B non-embedding). It features 36 transformer layers with grouped-query attention (32 Q heads, 8 KV heads) and supports a native context length of 32,768 tokens — extendable to 131,072 tokens via YaRN rope scaling. This MLX 4-bit quantization makes it practical for on-device use with `mlx_lm`.
Thinking & Non-Thinking Modes
A standout feature of Qwen3 is its ability to seamlessly switch between thinking mode and non-thinking mode within a single model. In thinking mode, the model performs step-by-step reasoning (wrapped in `<think>...</think>` blocks), ideal for math, coding, and complex logic. In non-thinking mode, it behaves more like a traditional instruct model — fast, direct, and efficient for general dialogue. Users can toggle between modes via the `enable_thinking` parameter or even mid-conversation using `/think` and `/no_think` tags.
Key Capabilities
- Strong reasoning in math, code generation, and commonsense logic — surpassing QwQ and Qwen2.5 instruct models in their respective domains
- Agent and tool-calling support, with precise external tool integration compatible with frameworks like Qwen-Agent and MCP servers
- Multilingual proficiency across 100+ languages and dialects, with strong translation and multilingual instruction-following
- Creative and conversational quality, including role-playing, multi-turn dialogues, and nuanced instruction following
Intended Use
Best suited for developers and researchers running inference locally on Apple Silicon Macs who need a capable, general-purpose LLM with advanced reasoning. Requires `mlx_lm ≥ 0.25.2` and `transformers ≥ 4.52.4`. Licensed under Apache 2.0.
spm https://github.com/trymirai/uzu.git
| 1 | import Uzu |
| 2 | |
| 3 | public func runChat() async throws { |
| 4 | let engineConfig = EngineConfig.create() |
| 5 | let engine = try await Engine.create(config: engineConfig) |
| 6 | |
| 7 | guard let model = try await engine.model(identifier: "Qwen/Qwen3-8B-MLX-4bit") else { |
| 8 | return |
| 9 | } |
| 10 | for try await update in try await engine.download(model: model).iterator() { |
| 11 | print("Download progress: \(update.progress())") |
| 12 | } |
| 13 | |
| 14 | let messages = [ |
| 15 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 16 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 17 | ] |
| 18 | let session = try await engine.chat(model: model, config: .create()) |
| 19 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 20 | var message: ChatMessage? = nil |
| 21 | for try await update in stream.iterator() { |
| 22 | switch update { |
| 23 | case .replies(let replies): |
| 24 | message = replies.last?.message |
| 25 | case .error(let error): |
| 26 | print("Error: \(error)") |
| 27 | } |
| 28 | } |
| 29 | print("Text: \(message?.text() ?? "empty")") |
| 30 | } |