Rapid-MLX: The Local Inference Server Your Mac Deserves

1. What Rapid-MLX Is: Stars, License, and the Idea

Rapid-MLX is an open-source, high-performance LLM server built exclusively for Apple Silicon Macs on top of Apple's MLX framework. One command serves an open model behind a drop-in OpenAI-compatible (and Anthropic-compatible) HTTP API, so any client that speaks the ChatGPT wire — agents, IDEs, or your own Python — can point at http://localhost:8000/v1 with no adapter and no API key. As of September 22, 2026 the repository at github.com/raullenchai/Rapid-MLX sits at roughly 3,800 stars and 418 forks under the Apache 2.0 license, with the catalog at 261 total aliases: 196 text, 11 image, 10 video, and 44 audio.

The core bet is hardware honesty: Ollama and llama.cpp are C++ engines with a Metal backend, while Rapid-MLX runs pure MLX kernels at native unified-memory bandwidth on M1 through M5. The repo claims 4.2x faster than Ollama in its tagline while the body documents up to 3x measured throughput — treat both as repo-reported numbers you can reproduce yourself with the built-in rapid-mlx benchmark run, which is exactly what the community leaderboard at rapidmlx.com is for.

2. Architecture in 4 Steps: From Request to Tokens

The README documents the engine as MLX-native serving with continuous batching, a quantized live KV cache, prompt caching, and speculative decoding. Here it is, step by step:

3. Quickstart: Serving Qwen in 60 Seconds (Verified)

Every command below comes straight from the live README and rapidmlx.com/download — I verified each one against the repo on September 22, 2026. Pick one install path, then serve. First run downloads the weights (~3 GB for the starter model) with a progress bar; after that it is seconds.

# Option A — Homebrew (prebuilt bottle, no tap needed)
brew install rapid-mlx

# Option B — guided installer (detects RAM, recommends a starter model)
curl -fsSL https://rapidmlx.com/install.sh | bash

# Option C — pip (needs Python 3.10+; macOS ships 3.9, so brew install python@3.12 first)
python3.12 -m pip install rapid-mlx
# Extras: pip install 'rapid-mlx[vision]'  |  pip install 'rapid-mlx[audio]'  |  pip install 'rapid-mlx[all]'

# Chat right now (defaults to qwen3.5-4b-4bit)
rapid-mlx chat

# Or serve it for every other app
rapid-mlx serve qwen3.5-4b-4bit
# => serving on http://localhost:8000/v1

# Sanity check the wire
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"default","messages":[{"role":"user","content":"Say hello"}]}'

# Point any OpenAI client at it — no key needed
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
print(client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Say hello"}],
).choices[0].message.content)

# Claude Code / Anthropic SDK uses the same host: http://localhost:8000
# (the Anthropic messages route lives at /v1/messages)

# Wire your coding agent with one command (server must be running)
rapid-mlx launch claude-code   # patches ~/.claude/settings.json, $0 per token
rapid-mlx launch list          # everything detected on this machine

# Self-diagnose when something feels off
rapid-mlx doctor               # metal, imports, CLI, model load: PASS/FAIL
rapid-mlx benchmark run qwen3.5-9b-4bit   # reproducible numbers for YOUR Mac

Pick the model by your RAM with rapid-mlx recipe or rapid-mlx info <alias> — Qwen3.5/Qwen3.6 default to thinking-on, so add --no-think if decode feels slow. One honest caveat from the README itself: Cursor routes BYOK requests through its own servers, which cannot reach your localhost — Cursor needs a public HTTPS tunnel plus RAPID_MLX_API_KEY, and that stops being fully local. Claude Code, Codex CLI, Aider, and OpenCode work against localhost directly.

4. Pick by RAM: What Fits Your Mac

Unified memory is the hard ceiling — the model, the KV cache, and macOS all share it. These are the repo-reported tiers and speeds (PyPI page, README tier map, and community benchmarks, September 2026):

The pattern across all four: match the alias to your unified memory first, benchmark second, and remember that single-request decode and 4-stream aggregate are different sports — the number that matters is the one measured in your workload, on your chip.

5. When NOT to Use Rapid-MLX: Honest Limits

I like Rapid-MLX, but it is a Mac-only server with opinions — it has a platform gate, a memory ceiling, and a setup step. Here is my honest checklist:

Conclusion

Rapid-MLX earns its ~3,800 stars honestly: it turns the Mac from a machine that can run models into a machine that serves them — MLX-native kernels, caches at every layer, speculative decoding, and an OpenAI-compatible surface that twelve agent CLIs already speak. The RAM-tier catalog plus the built-in benchmark harness make the usually painful “what fits my machine” question a two-command answer.

Install it with brew, serve qwen3.5-4b-4bit, point one agent at localhost:8000/v1, and compare a week of local tokens against your last API bill. If the quality holds for your workload, the server pays for itself the day you stop counting tokens.