Qwen/Qwen3-8B
Run locally on Apple devices with Mirai
- Type
- local
- From
- Alibaba
- Quantization
- No
- Parameters
- 8B
- Size
- 15.3 GB
- Source
Qwen3-8B is a dense 8.2 billion parameter causal language model from Alibaba's Qwen team, representing the latest generation in the Qwen series. Built on the Qwen3-8B-Base and refined through both pretraining and post-training, it introduces a distinctive dual-mode architecture that sets it apart from typical instruction-tuned models.
Thinking and Non-Thinking Modes
Qwen3-8B's headline feature is seamless switching between thinking mode and non-thinking mode within a single model. In thinking mode, the model engages in explicit chain-of-thought reasoning — ideal for complex math, code generation, and logical problem-solving — wrapping its internal reasoning in `<think>...</think>` blocks before delivering a final answer. In non-thinking mode, it operates as a streamlined conversational model akin to Qwen2.5-Instruct, prioritizing efficiency and directness. Users can toggle between modes via API parameters or even inline `/think` and `/no_think` commands within a conversation.
Architecture & Context
- Parameters: 8.2B total (6.95B non-embedding)
- Layers: 36, with Grouped Query Attention (32 Q heads, 8 KV heads)
- Context Length: 32,768 tokens natively, extensible to 131,072 tokens via YaRN RoPE scaling
- License: Apache 2.0
Key Capabilities
- Reasoning: Surpasses QwQ (thinking mode) and Qwen2.5-Instruct (non-thinking mode) on math, code, and commonsense reasoning benchmarks.
- Agent & Tool Use: Strong function-calling and tool integration abilities, compatible with MCP configurations and the Qwen-Agent framework for complex agentic workflows.
- Multilingual: Supports over 100 languages and dialects with robust multilingual instruction-following and translation.
- Human Preference Alignment: Excels in creative writing, role-playing, multi-turn dialogue, and instruction following.
Deployment
Qwen3-8B is compatible with Hugging Face Transformers (v4.51.0+), vLLM, SGLang, Ollama, LM Studio, llama.cpp, and other popular inference frameworks, making it straightforward to deploy as an OpenAI-compatible API or run locally.
spm https://github.com/trymirai/uzu.git
| 1 | import Uzu |
| 2 | |
| 3 | public func runChat() async throws { |
| 4 | let engineConfig = EngineConfig.create() |
| 5 | let engine = try await Engine.create(config: engineConfig) |
| 6 | |
| 7 | guard let model = try await engine.model(identifier: "Qwen/Qwen3-8B") else { |
| 8 | return |
| 9 | } |
| 10 | for try await update in try await engine.download(model: model).iterator() { |
| 11 | print("Download progress: \(update.progress())") |
| 12 | } |
| 13 | |
| 14 | let messages = [ |
| 15 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 16 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 17 | ] |
| 18 | let session = try await engine.chat(model: model, config: .create()) |
| 19 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 20 | var message: ChatMessage? = nil |
| 21 | for try await update in stream.iterator() { |
| 22 | switch update { |
| 23 | case .replies(let replies): |
| 24 | message = replies.last?.message |
| 25 | case .error(let error): |
| 26 | print("Error: \(error)") |
| 27 | } |
| 28 | } |
| 29 | print("Text: \(message?.text() ?? "empty")") |
| 30 | } |