Laya is the open alternative to hosted decision models: instead of generating text token by token, it answers typed questions — choice, score, true/false — in a single forward pass. And laya-mlx, an unofficial port released as v0.1.0 on September 19, 2026, runs it natively on Apple MLX.
No PyTorch, no Transformers runtime, no cloud API: 7.39 ms P50 with the multilingual checkpoint on an M3 Max, peaking at 687.6 MiB. Here is what it is, how it works in 4 steps, and how to run it in 5 minutes.
1. What It Is: Stars, License, Checkpoints
laya-mlx by mizorewww is an independent MLX reimplementation of Laya inference for Apple Silicon — not an official Convai Innovations release. Upstream Laya holds around 10.4k stars; the port passed 549 stars within a day of launch (September 20, 2026) and sits near 5.1k at the time of writing. License: Apache-2.0. Package v0.1.0 hit PyPI on September 19, 2026, and v0.2.0 is already published there, so plain pip install pulls the latest.
Repo
github.com/mizorewww/laya-mlx — unofficial port, Apache-2.0
Checkpoints (FP16, Hugging Face)
aac6fef/laya-mlx (421M EN, 512 tokens) · aac6fef/laya-multilingual-mlx (322M, 1,024 tokens) · aac6fef/laya-typed-decisions-mlx (421M, 1,024 tokens)
M3 Max numbers (project-reported)
P50 13.42 ms EN / 7.39 ms multilingual · batch-50: 146.8 q/s EN / 395.0 q/s multilingual · peak 943.6 MiB EN / 687.6 MiB multilingual
Fidelity
63/63 validation questions match upstream PyTorch on argmax in FP32 and FP16 (378/378 comparisons), per BENCHMARKS.md
Version note (read this first)
This post covers v0.1.0 (September 19, 2026) and every command below was verified against the repo and PyPI. Since v0.2.0 is already on PyPI, run pip install laya-mlx to get the latest — the install, import, and load calls are unchanged.
2. Architecture in 4 Steps
There is no text generation anywhere in this pipeline. You declare the question shape up front, and dedicated heads return calibrated probabilities. Zero output tokens is the whole pitch.
Step 1 — State + typed question in
You pass a state (text, JSON, or conversation) plus questions typed as choice (pick one label), score (ordered rubric levels), or noul (probability something is true).
Step 2 — Bidirectional encoder
One forward pass through ModernBERT-large (421M EN checkpoints) or mmBERT-base (322M multilingual). The model reads state and question together — no autoregressive decoding.
Step 3 — Decision stack in MLX
A learned decision Transformer plus scoring and action heads run entirely in MLX on Apple Silicon. Tokenization uses Hugging Face’s Rust tokenizer; PyTorch and the Transformers runtime are not needed for inference.
Step 4 — Calibrated probabilities out
You get answers with calibrated probabilities, four-decimal rounding, action probabilities, and token-usage fields — same schema as upstream, including fitted temperature calibration.
3. Quickstart (Verified)
You need an Apple Silicon Mac, macOS 14+, and Python 3.11+. The first load downloads the checkpoint; everything after that runs fully local.
pip install laya-mlx
import laya_mlx as laya
agent = laya.load("aac6fef/laya-mlx")
result = agent.predict(
"I was billed twice. Please refund the duplicate.",
{
"department": {
"type": "choice",
"instructions": "Who should handle this?",
"criteria": ["billing", "technical", "sales"],
}
},
)
print(result["answers"]["department"])The multilingual checkpoint (1024-token context, ~7 ms P50) loads the same way with aac6fef/laya-multilingual-mlx. The 843 MB English artifact downloads once; an independent Anthus benchmark reproduced the setup with make laya and confirmed the ~843 MB download.
What I checked before publishing
Repo URL, pip install laya-mlx, import laya_mlx as laya, load aac6fef/laya-mlx, Python 3.11+ and macOS 14+ requirements, and every benchmark figure — all verified against the repo README, BENCHMARKS.md, PyPI, and the HTX and Anthus write-ups. No flags invented: the snippet above is the README’s own minimal example.
4. Where It Shines: Use Cases
Think high-frequency, short-input, structured-output decisions — the layer between your app and an LLM, not a replacement for one.
Support triage
Route every ticket (billing vs technical vs sales) plus urgency score and refund intent in one ~13 ms call instead of a full LLM round trip.
Agent guardrails
Score or approve proposed agent actions with calibrated P(true) before executing — cheap enough to call on every step.
Real-time loops
The Snake demo re-evaluates every move locally at 75.40 moves/s with zero deaths across 2,400 moves — proof that per-frame decisions are viable.
Batch labeling
395 questions/s on the multilingual checkpoint makes large-scale classification and reranking practical on a single Mac.
5. When NOT to Use It
Small and fast does not mean universal. Skip laya-mlx when any of these apply:
- • You need generated text — summaries, chat, code. This model only returns typed decisions, never prose.
- • Your inputs are long documents. The English checkpoint caps at 512 tokens and the rest at 1,024 — state, instructions, and options share that budget, and overlong input truncates silently.
- • You are not on Apple Silicon. This is a native MLX runtime; on other hardware run upstream Laya on PyTorch instead.
- • Your choice questions have dozens of options. All labels share one token budget (Convai attributes its weak Banking77 result to exactly this), so shortlist to top-k first or split the question.
- • You want zero ops. A hosted API needs only a key; here you own the checkpoint, the runtime, and the calibration.
Golden rule
If your decision fits in a sentence and repeats thousands of times a day, laya-mlx is ideal. If it needs a paragraph of reasoning, call an LLM — or use laya-mlx as the fast filter in front of one.
6. Honest Limits
Every number in this post is project-reported on one machine: an M3 Max with a 40-core GPU and 128 GiB of unified memory. Treat them as a ceiling, not a promise — older Apple Silicon with fewer GPU cores will differ, and hosted CI only runs small-model CPU tests while GPU benchmarks are measured locally.
- • Context: 512 tokens EN, 1,024 multilingual and typed-decisions — questions and options included.
- • Multilingual quality is uneven across languages and may still need domain fine-tuning, per upstream docs.
- • Extra questions cost a forward pass each (~8 ms per added question on an M1 Max in the Anthus test) — unlike hosted models where questions ride in one request.
- • The port preserves upstream calibration and schema, but model quality limits remain those of the original checkpoints.
Reproducibility
Pin a Hub revision when it matters, validate on your own workload, and re-check calibration after any fine-tune — Anthus showed a fine-tuned Laya changes answers to unrelated questions on the same scorecard.
Conclusion
laya-mlx is the kind of repo that earns its stars: a focused, well-benchmarked port that turns a 421M-parameter decision model into a sub-15 ms local primitive for Mac apps. Single-digit milliseconds, under 1 GB, zero tokens, no cloud bill.
My verdict: if you build on Apple Silicon and your problem is decisions rather than prose, try it this week — pip install laya-mlx and the five-line snippet above is the whole evaluation. Just respect the 512/1,024-token budget and measure on your own chip before quoting the M3 Max numbers.
At a glance
- • laya-mlx v0.1.0 (Sep 19, 2026), Apache-2.0, unofficial MLX port
- • 7.39 ms P50 multilingual / 13.42 ms EN, 395 q/s batch, <1 GB peak
- • pip install laya-mlx → import laya_mlx → load aac6fef/laya-mlx
