Mirai Labs: Inference Engine

Run your model on 2 billion
Apple devices.
Perfect for:

Model companies.

You train and ship models. Mirai optimizes them for Apple Silicon, benchmarks on real hardware, and distributes.

AI researchers & labs.

Mirai converts your model and puts it in front of real users on Apple devices, not just leaderboards.

Independent makers.

You're fine-tuning or training from scratch. Mirai gives your model the same device reach as OpenAI and DeepSeek.

What Apple Silicon delivers today with Mirai.

Convert. Integrate. Run.

1

Choose framework

2

Run the following command to install Mirai SDK

https://github.com/trymirai/uzu-swift

3

Apply code

1import Foundation2import Uzu34public func runChat() async throws {5    let engineConfig = EngineConfig.create()6    let engine = try await Engine.create(config: engineConfig)7    8    guard let model = try await engine.model(identifier: "alibaba:qwen3.5:0.8b:mirai:mirai-m:4") else {9        return10    }11    for try await update in try await engine.download(model: model).iterator() {12        print(String(format: "\r\u{001B}[2KDownload progress: %.2f%%", update.progress() * 100), terminator: "")13        fflush(stdout)14    }15    print()16    17    let messages = [18        ChatMessage.system().withText(text: "You are a helpful assistant"),19        ChatMessage.user().withText(text: "Tell me a short, funny story about a robot")20    ]21    let session = try await engine.chat(model: model, config: .create())22    let stream = await session.replyWithStream(input: messages, config: .create())23    var message: ChatMessage? = nil24    for try await update in stream.iterator() {25        switch update {26        case .replies(let replies):27            let reply = replies.last28            message = reply?.message29            print("Generated tokens: \(reply?.stats.tokensCountOutput ?? 0)")30        case .error(let error):31            print("Error: \(error)")32        }33    }34    print("Reasoning: \(message?.reasoning() ?? "empty")")35    print("Text: \(message?.text() ?? "empty")")36}37

One inference engine. Integrate from any language.

RustCargo

cargo add uzu --git https://github.com/trymirai/uzu
cargo add tokio --features full

SwiftSwift Package Manager

https://github.com/trymirai/uzu-swift

TypeScriptNPM (Node.js)

pnpm add typescript ts-node @types/node -D
pnpm add @trymirai/uzu

PythonPyPI

uv add uzu

KotlinComing Soon

Learn more

Built-in features every model gets automatically:

Speculative decoding

Structured output

Task-specific sessions

Built-in performance metrics

swift

Supported models:

Your model can be nextTalk to us

Explore all models

Common questions:

How does model support work?

Our stack consists of two main products:

A model conversion & optimization toolkit

It takes original model checkpoints and converts them into a format supported by the inference engine. This format is an intermediate representation composed of unified blocks, each backed by a corresponding encoded sequence in the inference engine. If your model is built using already supported blocks, nothing special is required, you can simply run the conversion command yourself. If your model introduces custom layers, we can easily add support for them (and you’re welcome to implement it yourself as well).

The inference engine

Each unique block from the intermediate representation has corresponding backend-specific kernels, which are executed on the target compute device during inference. Similar to the conversion toolkit, if your model introduces a new type of layer, the corresponding kernels will need to be implemented.

What architectures are supported?

We currently support a wide range of architectures. You can find the full list of supported models here:

Full list of local models

How does Mirai compare to other inference engines?

As shown in our benchmarks, Mirai significantly outperforms other inference engines on edge devices. The performance boost varies depending on the specific model-device pairs. We also provide out-of-the-box speculative decoding. In this setup, proposals are generated using a very small model (<50MB), allowing it to run even on iOS devices.

What is the maximum supported model size?

This fully depends on device capabilities. On macOS, very large models such as GPT-OSS can run depending on available memory. On iOS, we recommend staying under 3GB of RAM usage. Quantized 4B models are typically a great fit.

How can I run benchmarks myself?

Our inference engine includes a built-in CLI tool with a dedicated benchmark command. You can find more details here:

Link to the CLI tool

How can we discuss a specific use case?

Feel free to reach out via email.

contact@trymirai.com

You can also join our Discord:

Mirai Discord

Or book a call with our engineers. We’d be happy to discuss your specific requirements:

Book a call