mlx-community/Llama-3.2-1B-Instruct-4bit
Run locally on Apple devices with Mirai
- Type
- local
- From
- Meta
- Quantization
- MLX 4-bit
- Parameters
- 1B
- Size
- 679.7 MB
- Source
A 4-bit quantized version of Meta's Llama 3.2 1B Instruct model, optimized for Apple Silicon via the MLX framework. Published by the MLX Community, this conversion brings Meta's compact instruction-tuned language model to Mac-native inference with significantly reduced memory requirements.
Origin & Architecture
Llama 3.2 1B is part of Meta's September 2024 Llama 3.2 release — a family of efficient, smaller-scale language models designed for accessible deployment. The 1B-parameter instruct variant has been fine-tuned for conversational and instruction-following tasks. This community conversion applies 4-bit quantization, shrinking the model's footprint while preserving practical generation quality — ideal for on-device use on MacBooks and other Apple Silicon hardware.
Multilingual Support
The model supports eight languages out of the box: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai, making it a versatile option for multilingual text generation and chat applications.
Key Strengths
- Ultra-lightweight: At 1B parameters with 4-bit quantization, the model runs comfortably on devices with limited memory, including entry-level Apple Silicon Macs.
- MLX-native: Purpose-built for the MLX ecosystem, enabling fast, efficient inference without external GPU dependencies.
- Instruction-tuned: Aligned for assistant-style dialogue, summarization, question answering, and general instruction following.
- Low barrier to entry: Well suited for prototyping, local development, edge deployment, and privacy-sensitive use cases where cloud inference is undesirable.
Use Cases
This model is a strong fit for developers building local AI assistants, lightweight chatbots, or multilingual text utilities on macOS. Its small size makes it especially appealing for experimentation, rapid iteration, and embedded applications where latency and resource constraints matter more than peak benchmark performance.
Licensed under the Llama 3.2 Community License.
spm https://github.com/trymirai/uzu.git
| 1 | import Uzu |
| 2 | |
| 3 | public func runChat() async throws { |
| 4 | let engineConfig = EngineConfig.create() |
| 5 | let engine = try await Engine.create(config: engineConfig) |
| 6 | |
| 7 | guard let model = try await engine.model(identifier: "mlx-community/Llama-3.2-1B-Instruct-4bit") else { |
| 8 | return |
| 9 | } |
| 10 | for try await update in try await engine.download(model: model).iterator() { |
| 11 | print("Download progress: \(update.progress())") |
| 12 | } |
| 13 | |
| 14 | let messages = [ |
| 15 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 16 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 17 | ] |
| 18 | let session = try await engine.chat(model: model, config: .create()) |
| 19 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 20 | var message: ChatMessage? = nil |
| 21 | for try await update in stream.iterator() { |
| 22 | switch update { |
| 23 | case .replies(let replies): |
| 24 | message = replies.last?.message |
| 25 | case .error(let error): |
| 26 | print("Error: \(error)") |
| 27 | } |
| 28 | } |
| 29 | print("Text: \(message?.text() ?? "empty")") |
| 30 | } |