mlx-community/Llama-3.2-1B-Instruct-8bit
Run locally on Apple devices with Mirai
- Type
- local
- From
- Meta
- Quantization
- MLX 8-bit
- Parameters
- 1B
- Size
- 1.2 GB
- Source
A compact, instruction-tuned language model from Meta's Llama 3.2 family, converted to 8-bit quantization for Apple's MLX framework. Published by the mlx-community, this variant makes it straightforward to run Llama 3.2 1B Instruct efficiently on Apple Silicon hardware with minimal memory overhead.
Model Overview
Llama 3.2 1B Instruct is one of Meta's smallest Llama 3.2 releases, designed for lightweight text generation tasks where low latency and resource efficiency are priorities. The 8-bit quantized MLX conversion reduces the model's memory footprint while preserving the quality of the original instruction-tuned weights, making it well-suited for local inference on MacBooks, Mac Minis, and other Apple Silicon devices.
Multilingual Capabilities
The model supports eight languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai — enabling multilingual chat, translation assistance, and cross-lingual text generation out of the box.
Key Strengths
- Ultra-lightweight: At 1 billion parameters with 8-bit quantization, this is one of the most resource-friendly instruct models in the Llama 3 ecosystem.
- Apple Silicon optimized: Native MLX format ensures efficient memory usage and fast inference on M-series chips.
- Instruction-following: Fine-tuned for conversational and task-oriented prompts, suitable for assistants, summarization, Q&A, and simple reasoning.
- Broad language support: Eight languages covered natively, broadening accessibility for international applications.
Use Cases
This model is a strong fit for on-device chatbots, rapid prototyping, edge deployments, and scenarios where privacy-sensitive data should not leave the local machine. Its small size also makes it useful as a fast draft model in speculative decoding pipelines or as a lightweight component in agentic workflows.
Provenance
Based on Meta's Llama 3.2, released September 25, 2024. Quantized and converted to MLX format by the mlx-community. Licensed under the Llama 3.2 Community License.
spm https://github.com/trymirai/uzu.git
| 1 | import Uzu |
| 2 | |
| 3 | public func runChat() async throws { |
| 4 | let engineConfig = EngineConfig.create() |
| 5 | let engine = try await Engine.create(config: engineConfig) |
| 6 | |
| 7 | guard let model = try await engine.model(identifier: "mlx-community/Llama-3.2-1B-Instruct-8bit") else { |
| 8 | return |
| 9 | } |
| 10 | for try await update in try await engine.download(model: model).iterator() { |
| 11 | print("Download progress: \(update.progress())") |
| 12 | } |
| 13 | |
| 14 | let messages = [ |
| 15 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 16 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 17 | ] |
| 18 | let session = try await engine.chat(model: model, config: .create()) |
| 19 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 20 | var message: ChatMessage? = nil |
| 21 | for try await update in stream.iterator() { |
| 22 | switch update { |
| 23 | case .replies(let replies): |
| 24 | message = replies.last?.message |
| 25 | case .error(let error): |
| 26 | print("Error: \(error)") |
| 27 | } |
| 28 | } |
| 29 | print("Text: \(message?.text() ?? "empty")") |
| 30 | } |