If you own an Apple Silicon Mac, you are sitting on an inference server and you may not know it. vMLX — github.com/jjang-ai/vmlx, ~860 stars and Apache 2.0 — is the most complete MLX inference engine for the Mac: one command serves any mlx-community model behind an OpenAI, Anthropic and Ollama compatible API.
In this guide I show you what it is, how its 4-layer performance architecture works, how to run it in 5 minutes with vmlx serve, and — just as important — when you should NOT use it.
Repo Fact Sheet
Repo
jjang-ai/vmlx
Stars
~860 (Sep 2026)
License
Apache 2.0
Install
pip install vmlx
API
OpenAI + Anthropic + Ollama
App
MLX Studio (vmlx.net)
1. What vMLX Is (and What It Is Not)
vMLX is a self-hosted inference server for LLMs, VLMs and image generation on Apple Silicon, built on Apple’s MLX framework with Metal GPU acceleration. You point it at any Hugging Face mlx-community repo — Qwen, Llama 4, Gemma 4, DeepSeek V4, Kimi, Mistral, Nemotron-3-Omni — and it serves it at http://localhost:8000 with the same wire format as OpenAI. No cloud, no API keys, nothing leaves your machine.
What it is not: it is not a training framework, not a CUDA server, and not cross-platform. It only runs on macOS with Apple Silicon (M1–M4) and Python 3.11+. If your fleet is NVIDIA, you want vLLM or SGLang — I compare those two in a related post linked below.
One Server, Three APIs
The same port speaks POST /v1/chat/completions (OpenAI), POST /v1/messages (Anthropic) and POST /api/chat (Ollama), plus embeddings, rerank, Whisper STT and Kokoro TTS endpoints. Point the OpenAI, Anthropic or ollama CLI at it unchanged — even OLLAMA_HOST=http://localhost:8080 ollama run works through the desktop gateway.
2. The Architecture in 4 Steps
Every request flows through the same pipeline. Understanding these four layers is what separates “it runs” from “it flies”:
Step 1 — MLX-native engine
unified memory · Metal kernels
The Python/FastAPI engine dispatches to mlx-lm (text), mlx-vlm (vision), mflux (Flux image gen) and mlx-audio (speech). Weights, activations and KV cache all live in unified memory — no CPU↔GPU copies.
Step 2 — Continuous batching
flag: --continuous-batching
Multiple concurrent requests share one forward pass instead of queueing behind each other. This is the flag that makes the server multi-user instead of a demo.
vmlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --continuous-batching
Step 3 — 5-layer cache stack
prefix · paged · disk L2 · KV quant
L1 prefix/paged KV cache reuses prefill work (the project reports up to ~9.7x faster time-to-first-token on cached prompts, M3 Ultra, Feb 2026); L2 disk cache survives restarts; q4/q8 KV quantization compresses stored states 2–4x.
vmlx serve mlx-community/Llama-3.2-3B-Instruct-4bit --continuous-batching --enable-prefix-cache --use-paged-cache
Step 4 — Fast decoding
--enable-pld · speculative draft
Prompt-Lookup Decoding reuses n-gram matches from the prompt — no draft model needed, ideal for code and JSON. Classic speculative decoding with a small draft model gives 20–90% speedups on supported models.
vmlx serve mlx-community/Qwen3-8B-4bit --continuous-batching --enable-pld
The Two Multipliers: JANG + Multi-Mac
JANG adaptive mixed-precision assigns different bit widths per layer type (attention 8-bit, MLP 2–3 bit) — JANG_3M (~3.2 avg bits) is the recommended profile, and pre-quantized models live under JANGQ-AI on Hugging Face. For models that exceed one Mac, pipeline parallelism splits layers across machines over Thunderbolt/Ethernet: workers run vmlx-worker --secret, the coordinator runs vmlx serve <model> --distributed --cluster-secret.
3. Quickstart: From Zero to Server in 5 Minutes
Requirements, verified from the docs: macOS on Apple Silicon, Python 3.11+, 8 GB RAM minimum (16 GB+ recommended). Install takes one of three documented paths:
Install (pick ONE — uv is recommended)
# Recommended: uv (no venv hassle) brew install uv uv tool install vmlx # Or: pipx (isolated from system Python) brew install pipx pipx install vmlx # Or: pip inside a virtual environment python3 -m venv ~/.vmlx-env && source ~/.vmlx-env/bin/activate pip install vmlx
Serve your first model
vmlx serve mlx-community/Qwen3-8B-4bit # Server live at http://0.0.0.0:8000 (OpenAI + Anthropic API)
Call it with the OpenAI SDK (unchanged code)
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
model="local",
messages=[{"role": "user", "content": "Hello!"}],
stream=True,
)
for chunk in resp:
print(chunk.choices[0].delta.content or "", end="", flush=True)Quantize with JANG (recommended profile JANG_3M)
pip install "vmlx[jang]" vmlx convert my-model --jang-profile JANG_3M vmlx serve ./my-model-JANG_3M --continuous-batching --use-paged-cache
Image + audio extras (optional dependencies)
pip install "vmlx[image]" # Flux schnell/dev via mflux vmlx serve schnell pip install mlx-audio # Kokoro TTS + Whisper STT vmlx serve kokoro --port 8002
The macOS 14+ venv Gotcha
On macOS 14+, bare pip install fails with “externally-managed-environment”. That is not a vMLX bug — it is Apple/Python PEP 668 protection. Use uv, pipx, or a venv as shown above. Prefer the desktop app (MLX Studio, signed DMG for Tahoe/Sequoia) if you never want to touch a terminal: it bundles its own Python 3.12.
Verify It Works
curl http://localhost:8000/health for the server, GET /v1/models to list loaded models, and GET /v1/cache/stats to confirm prefix hits. If a model OOMs, drop to a smaller quantized build (e.g. Llama-3.2-1B-Instruct-4bit) before touching any flag.
4. Real-World Use Cases
vMLX shines wherever a Mac is already on the desk. These are the four cases where I would reach for it first:
Local dev loop for agents
Claude Code, OpenCode, Continue or Cursor pointed at localhost:8000 give you a zero-cost, offline model backend for tool-calling and structured output while you iterate.
Home-lab OpenAI drop-in
One always-on Mac serving the family of mlx-community models to every script, notebook and side project on your LAN — same SDK code as production OpenAI.
Two-Mac cluster for giant models
Pipeline parallelism over Thunderbolt lets two or three Macs jointly serve MoE models (e.g. ~400B-class JANG builds) that fit on no single machine.
Multimodal corner: image + voice
Flux Schnell text-to-image, Qwen instruction-based image edits, Kokoro TTS voices (Spanish included) and Whisper STT — all from the same server process.
5. The Honest Part: macMLX, and When NOT to Use vMLX
No tool review is complete without its limits. Here is the contrast and the red lines, all sourced from the projects’ own docs:
One Paragraph on macMLX (the Swift-native alternative)
If vMLX’s Python/FastAPI core bothers you, macMLX (macmlx.app, Apache 2.0) is the native counterpoint: inference engine written in Swift running in-process, ~50 MB app bundle, zero Python runtime, always-on OpenAI-compatible API on macOS 14+ Apple Silicon. Rule of thumb: vMLX for the broadest model coverage and Python extensibility (JANG, mflux, mlx-audio ecosystem); macMLX for the leanest native footprint on a single Mac.
⛔ Do NOT use vMLX when…
- • You are on Linux, Windows or NVIDIA GPUs — vMLX is Apple Silicon only. Use vLLM or SGLang there.
- • You need CUDA multi-GPU tensor parallelism at datacenter scale — that is vLLM/SGLang territory, not a Mac server.
- • You want Smelt partial-expert loading AND vision input together — Smelt auto-disables VLM mode (vision on a Smelt model produces garbage logits, per the docs).
- • Your image-edit model needs ~54 GB (Qwen Image Edit) but your Mac has 8–16 GB unified memory — pick Flux Schnell 4-bit instead.
- • You need a notarized one-click app and fear the terminal — get the MLX Studio DMG rather than fighting pip.
Golden rule
If the workload fits in one Mac’s unified memory and speaks HTTP, vMLX is the fastest path from model to API. The moment you need CUDA, Windows, or datacenter scale, you have outgrown it — and that is fine, it was never that tool.
Conclusion
vMLX turns the Mac from a client into infrastructure: OpenAI-compatible serving, production-grade batching and caching, JANG quantization that punches above its bit-width, and multi-Mac clusters over a Thunderbolt cable. For Apple Silicon owners, it is the closest thing to a private inference cloud you can apt-install — sorry, uv-install — in an afternoon.
Start with uv tool install vmlx plus one 8B 4-bit model, confirm /health and /v1/cache/stats, then decide whether you need JANG or a second Mac. And if your fleet is NVIDIA instead, read the vLLM vs SGLang guide next.
vMLX: Recap
Run
- • uv tool install vmlx
- • vmlx serve <model>
- • localhost:8000/v1
Speed up
- • --continuous-batching
- • --enable-prefix-cache
- • --enable-pld / JANG_3M
Scale / skip
- • vmlx-worker clusters
- • MLX Studio DMG
- • Not for CUDA/NVIDIA
Sources: github.com/jjang-ai/vmlx · vmlx.net · PyPI vmlx · macmlx.app · MLX Studio
