Qwen/Qwen3-32B-MLX-4bit
Run locally on Apple devices with Mirai
- Type
- local
- From
- Alibaba
- Quantization
- MLX 4-bit
- Parameters
- 32B
- Size
- 16.2 GB
- Source
A 4-bit quantized version of Qwen3-32B, optimized for efficient inference on Apple Silicon using the MLX framework. This model is part of the Qwen3 generation of large language models from the Qwen team at Alibaba, delivering strong reasoning, multilingual, and agentic capabilities in a format tailored for local Mac-based workflows.
Architecture & Specs
Qwen3-32B is a 32.8B-parameter dense causal language model with 64 layers, grouped-query attention (64 Q heads, 8 KV heads), and a native context length of 32,768 tokens — extensible to 131,072 tokens via YaRN rope scaling. The MLX 4-bit variant significantly reduces memory requirements, making it practical to run on high-end Apple Silicon devices.
Key Capabilities
- Dual-mode reasoning: Seamlessly switches between a thinking mode (step-by-step reasoning for math, code, and logic) and a non-thinking mode (fast, general-purpose dialogue) within a single model. Users can toggle modes via `enable_thinking` or inline `/think` and `/no_think` tags mid-conversation.
- Strong reasoning performance: Surpasses prior QwQ (thinking mode) and Qwen2.5 Instruct (non-thinking mode) on mathematics, code generation, and commonsense reasoning benchmarks.
- Agent & tool use: Excellent function-calling and tool integration, compatible with frameworks like Qwen-Agent and MCP server configurations for agentic workflows.
- Multilingual: Supports 100+ languages and dialects, with strong instruction-following and translation quality.
- Human preference alignment: Excels in creative writing, role-playing, multi-turn dialogue, and nuanced instruction following.
Intended Use
This model is ideal for developers and researchers running local inference on Mac hardware who need a powerful, general-purpose LLM with advanced reasoning. It requires `mlx_lm ≥ 0.25.2` and `transformers ≥ 4.52.4`. Licensed under Apache 2.0.
spm https://github.com/trymirai/uzu.git
| 1 | import Uzu |
| 2 | |
| 3 | public func runChat() async throws { |
| 4 | let engineConfig = EngineConfig.create() |
| 5 | let engine = try await Engine.create(config: engineConfig) |
| 6 | |
| 7 | guard let model = try await engine.model(identifier: "Qwen/Qwen3-32B-MLX-4bit") else { |
| 8 | return |
| 9 | } |
| 10 | for try await update in try await engine.download(model: model).iterator() { |
| 11 | print("Download progress: \(update.progress())") |
| 12 | } |
| 13 | |
| 14 | let messages = [ |
| 15 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 16 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 17 | ] |
| 18 | let session = try await engine.chat(model: model, config: .create()) |
| 19 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 20 | var message: ChatMessage? = nil |
| 21 | for try await update in stream.iterator() { |
| 22 | switch update { |
| 23 | case .replies(let replies): |
| 24 | message = replies.last?.message |
| 25 | case .error(let error): |
| 26 | print("Error: \(error)") |
| 27 | } |
| 28 | } |
| 29 | print("Text: \(message?.text() ?? "empty")") |
| 30 | } |