mlx-community/gemma-3-4b-it-8bit
Run locally on Apple devices with Mirai
- Type
- local
- From
- Quantization
- MLX 8-bit
- Parameters
- 4B
- Size
- 3.9 GB
- Source
An 8-bit quantized version of Google's Gemma 3 4B Instruct model, converted to the MLX format for efficient inference on Apple Silicon hardware. This community conversion was produced using `mlx-vlm` and is based on the official `google/gemma-3-4b-it` release.
Overview
Gemma 3 4B IT is a compact, instruction-tuned language model from Google's Gemma family. It supports multimodal image-text-to-text tasks, meaning it can accept both images and text as input and generate text responses — making it suitable for visual question answering, image captioning, and general conversational use.
The MLX conversion brings this model natively to Apple's MLX framework, enabling fast on-device inference on Mac computers equipped with M-series chips. The 8-bit quantization reduces memory usage compared to the full-precision model while retaining strong output quality.
Key Details
- Base model: Google Gemma 3 4B PT (pretrained), instruction-tuned variant
- Architecture: Gemma 3 (transformer-based, multimodal)
- Quantization: 8-bit (MLX format)
- Pipeline: Image-text-to-text generation
- Framework: MLX / mlx-vlm
- License: Gemma (Google usage license, gated access required)
Use Cases
- Visual understanding: Describe, analyze, or answer questions about images
- Conversational AI: Lightweight instruction-following chat on local hardware
- Edge deployment: Run a capable multimodal model entirely on-device with Apple Silicon, no cloud dependency needed
Notes
This is a community-maintained conversion. For full model documentation, training details, and evaluation benchmarks, refer to the original Gemma 3 4B IT model card. Access requires agreeing to Google's Gemma usage license via Hugging Face.
spm https://github.com/trymirai/uzu.git
| 1 | import Uzu |
| 2 | |
| 3 | public func runChat() async throws { |
| 4 | let engineConfig = EngineConfig.create() |
| 5 | let engine = try await Engine.create(config: engineConfig) |
| 6 | |
| 7 | guard let model = try await engine.model(identifier: "mlx-community/gemma-3-4b-it-8bit") else { |
| 8 | return |
| 9 | } |
| 10 | for try await update in try await engine.download(model: model).iterator() { |
| 11 | print("Download progress: \(update.progress())") |
| 12 | } |
| 13 | |
| 14 | let messages = [ |
| 15 | ChatMessage.system().withText(text: "You are a helpful assistant"), |
| 16 | ChatMessage.user().withText(text: "Tell me a short, funny story about a robot") |
| 17 | ] |
| 18 | let session = try await engine.chat(model: model, config: .create()) |
| 19 | let stream = await session.replyWithStream(input: messages, config: .create()) |
| 20 | var message: ChatMessage? = nil |
| 21 | for try await update in stream.iterator() { |
| 22 | switch update { |
| 23 | case .replies(let replies): |
| 24 | message = replies.last?.message |
| 25 | case .error(let error): |
| 26 | print("Error: \(error)") |
| 27 | } |
| 28 | } |
| 29 | print("Text: \(message?.text() ?? "empty")") |
| 30 | } |