Metrics
Benchmarked 3 Sept 2026
We compare uzu, MLX, and llama.cpp locally on the same Apple device.
How we measureOutput speedThe speed at which text appears on screen We use a fixed 1,355 token prompt to measure autoregressive output speed, input speed and resident memory consumption. Each score is computed as a mean of three consecutive runs. Speculative decoding output speed is computed as a macro-average of measurements collected on random samples from three datasets with default sampling parameters: - MT-Bench — conversational prompts, 80 samples - MATH-500 — mathematical reasoning, 128 samples - HumanEval — code generation, 128 samples
uzu is the fastest in 12 of 12 models on this device.
tok/sHigher is better
LFM2.51.2B Instruct · 4bit
LFM2.51.2B Thinking · 4bit
LFM2.51.2B Instruct · 8bit
LFM2.51.2B Thinking · 8bit
Qwen 3.52B · 4bit
LFM2.52.6B · 4bit
Qwen 3.52B · 8bit
LFM2.52.6B · 8bit
Qwen 3.54B · 4bit
Qwen 3.59B · 4bit
Qwen 3.54B · 8bit
Qwen 3.59B · 8bit
For uzu, we use Mirai-M, Mirai-L as 4-bit and 8-bit references. For llama.cpp we select the closest quantized checkpoints based on KL divergence as measured by our KL eval tool. For mlx we always use mlx-community checkpoints.
Input speedHow fast the model processes the prompt Uzu uses native prefill_tokens_per_second; MTPLX uses prompt_tps; llama.cpp uses prompt_per_second; MLX divides prompt-token count by wall-clock time to the first token.
uzu is the fastest in 11 of 12 models on this device.
tok/sHigher is betterLFM2.51.2B Instruct · 8bit
LFM2.51.2B Thinking · 8bit
LFM2.51.2B Instruct · 4bit
LFM2.51.2B Thinking · 4bit
Qwen 3.52B · 8bit
Qwen 3.52B · 4bit
LFM2.52.6B · 8bit
LFM2.52.6B · 4bit
Qwen 3.54B · 8bit
Qwen 3.54B · 4bit
Qwen 3.59B · 8bit
Qwen 3.59B · 4bit
For uzu, we use Mirai-M, Mirai-L as 4-bit and 8-bit references. For llama.cpp we select the closest quantized checkpoints based on KL divergence as measured by our KL eval tool. For mlx we always use mlx-community checkpoints.
Resident memoryHow much RAM the model occupies while running Uzu reports session.peak_memory_usage; MLX and MTPLX use native peak-memory counters; llama.cpp uses process resident memory.
uzu leads in 12 of 12 models on this device.
GiBLower is betterLFM2.51.2B Instruct · 4bit
LFM2.51.2B Thinking · 4bit
Qwen 3.52B · 4bit
LFM2.51.2B Instruct · 8bit
LFM2.51.2B Thinking · 8bit
LFM2.52.6B · 4bit
Qwen 3.52B · 8bit
Qwen 3.54B · 4bit
LFM2.52.6B · 8bit
Qwen 3.54B · 8bit
Qwen 3.59B · 4bit
Qwen 3.59B · 8bit
For uzu, we use Mirai-M, Mirai-L as 4-bit and 8-bit references. For llama.cpp we select the closest quantized checkpoints based on KL divergence as measured by our KL eval tool. For mlx we always use mlx-community checkpoints.
Quantization quality
How we measure
Each configuration is run three times using the same prompt, which is approximately 1,357 input tokens long (the exact number may vary depending on the tokenizer and chat template). The maximum output length is set to 4,096 tokens. Before each iteration, the runner waits until both CPU and GPU temperatures are below 60°C.
uzu is run via its CLI. MLX is run through mlx-lm. llama.cpp is run through llama-cpp-python for autoregressive runs and through the llama.cpp server for speculative decoding runs. All engines use the same sampling parameters where supported, otherwise, they use their respective client defaults.