mlx-community/Llama-3.2-3B-Instruct-4bit
Run locally on Apple devices with Mirai
- Type
- local
- From
- Meta
- Quantization
- MLX 4-bit
- Parameters
- 3B
- Size
- 1.7 GB
- Source
A 4-bit quantized version of Meta's Llama 3.2 3B Instruct model, optimized for Apple Silicon via the MLX framework. Published by the MLX Community, this conversion brings the compact yet capable Llama 3.2 instruction-tuned model to Mac-native inference with significantly reduced memory requirements.
What It Is
Llama 3.2 3B Instruct is a 3-billion-parameter language model from Meta, fine-tuned for instruction-following and conversational tasks. This variant applies 4-bit quantization through the MLX ecosystem, making it practical to run locally on MacBooks and other Apple Silicon devices without a dedicated GPU server.
The base model supports eight languages — English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai — giving it broad multilingual utility even at a compact size.
Key Highlights
- Architecture: Llama 3.2, part of Meta's third-generation Llama family
- Parameters: 3 billion (4-bit quantized)
- Framework: MLX — Apple's machine learning framework built for M-series chips
- Task: Text generation, instruction following, and dialogue
- Multilingual: Covers eight languages out of the box
Best For
This model is well suited for developers and researchers who want a lightweight, responsive local language model on macOS. Typical use cases include:
- Local chatbots and assistants
- Multilingual text generation and summarization
- Prototyping and experimentation without cloud inference costs
- On-device applications where privacy or latency matters
Provenance
The base model was released by Meta on September 25, 2024, under the Llama 3.2 Community License. The MLX Community performed the 4-bit quantization and conversion for seamless use with the `mlx-lm` toolchain and compatible applications.
spm https://github.com/trymirai/uzu.git
| 1 | import Uzu |
| 2 | |
| 3 | public func runChat() async throws { |
| 4 | let engineConfig = EngineConfig.create() |
| 5 | let engine = try await Engine.create(config: engineConfig) |
| 6 | |
| 7 | guard let model = try await engine.model(identifier: "mlx-community/Llama-3.2-3B-Instruct-4bit") else { |
| 8 | return |
| 9 | } |
| 10 | for try await update in try await engine.download(model: model).iterator() { |
| 11 | print("Download progress: \(update.progress())") |
| 12 | } |
| 13 | |
| 14 | let messages = [ |
| 15 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 16 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 17 | ] |
| 18 | let session = try await engine.chat(model: model, config: .create()) |
| 19 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 20 | var message: ChatMessage? = nil |
| 21 | for try await update in stream.iterator() { |
| 22 | switch update { |
| 23 | case .replies(let replies): |
| 24 | message = replies.last?.message |
| 25 | case .error(let error): |
| 26 | print("Error: \(error)") |
| 27 | } |
| 28 | } |
| 29 | print("Text: \(message?.text() ?? "empty")") |
| 30 | } |