Qwen/Qwen3-8B-MLX-8bit
Run locally on Apple devices with Mirai
- Type
- local
- From
- Alibaba
- Quantization
- MLX 8-bit
- Parameters
- 8B
- Size
- 7.9 GB
- Source
An 8-bit quantized version of Qwen3-8B, optimized for Apple Silicon via the MLX framework. Part of the Qwen3 generation of large language models from Alibaba's Qwen team, this variant brings efficient on-device inference to Mac users without sacrificing the core capabilities of the full-precision model.
Architecture & Specs
Qwen3-8B is a causal language model with 8.2 billion parameters (6.95B non-embedding), featuring 36 layers and grouped-query attention (32 Q heads, 8 KV heads). It supports a native context length of 32,768 tokens, extensible to 131,072 tokens using YaRN rope scaling.
Key Capabilities
Dual-mode reasoning is a standout feature of Qwen3. The model supports seamless switching between a *thinking mode* — where it performs step-by-step reasoning wrapped in `<think>...</think>` blocks — and a *non-thinking mode* for fast, general-purpose dialogue. Users can toggle between modes via the `enable_thinking` parameter or inline `/think` and `/no_think` tags within conversations.
Beyond reasoning, Qwen3-8B delivers strong performance across:
- Code generation and mathematics, leveraging its deep reasoning capabilities
- Creative writing, role-playing, and multi-turn dialogue with improved human preference alignment
- Agentic workflows and tool calling, with precise external tool integration in both thinking modes
- Multilingual support across 100+ languages and dialects, including translation and multilingual instruction following
Usage
This model is designed for use with the `mlx_lm` Python package (≥ 0.25.2) and is compatible with `transformers` (≥ 4.52.4). It works well for local inference on Apple Silicon Macs, offering a practical balance of quality and efficiency through 8-bit quantization.
Provenance
Released under the Apache 2.0 license by the Qwen team. For benchmarks, detailed training methodology, and deployment guidance, refer to the official Qwen3 blog and documentation.
spm https://github.com/trymirai/uzu.git
| 1 | import Uzu |
| 2 | |
| 3 | public func runChat() async throws { |
| 4 | let engineConfig = EngineConfig.create() |
| 5 | let engine = try await Engine.create(config: engineConfig) |
| 6 | |
| 7 | guard let model = try await engine.model(identifier: "Qwen/Qwen3-8B-MLX-8bit") else { |
| 8 | return |
| 9 | } |
| 10 | for try await update in try await engine.download(model: model).iterator() { |
| 11 | print("Download progress: \(update.progress())") |
| 12 | } |
| 13 | |
| 14 | let messages = [ |
| 15 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 16 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 17 | ] |
| 18 | let session = try await engine.chat(model: model, config: .create()) |
| 19 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 20 | var message: ChatMessage? = nil |
| 21 | for try await update in stream.iterator() { |
| 22 | switch update { |
| 23 | case .replies(let replies): |
| 24 | message = replies.last?.message |
| 25 | case .error(let error): |
| 26 | print("Error: \(error)") |
| 27 | } |
| 28 | } |
| 29 | print("Text: \(message?.text() ?? "empty")") |
| 30 | } |