Running a capable LLM on your own Mac used to mean fighting VRAM limits, CUDA-only tooling, or paying for cloud GPUs by the hour. Apple’s MLX stack changes that equation: it treats the Mac’s unified memory as one pool, speaks Metal natively, and with the mlx-lm package you can generate, quantize, and fine-tune models with a handful of commands.
In this deep-dive I show you the full loop: install mlx-lm, chat with the default Llama-3.2-3B, convert and 4-bit quantize any Hugging Face model in seconds, fine-tune it with LoRA on top of the quantized weights, and publish to mlx-community. Plus real M5 vs M4 numbers from Apple’s September 2026 research post.
1. The Problem: Great Mac, No Good Way to Run LLMs
Most open LLM tooling was designed around NVIDIA GPUs: discrete VRAM, CUDA kernels, and servers that never sleep. A MacBook with 18GB or 36GB of unified memory looks weak on paper next to an 80GB H100, yet in practice the Mac can hold a 14B model in memory while the cloud bill keeps ticking. The missing piece was software that actually uses the Apple Silicon architecture instead of emulating the CUDA world.
The Most Common Mistake
Installing a generic x86 inference stack on a Mac and concluding “Apple Silicon is too slow for LLMs”. Without unified-memory-aware execution and Metal kernels, you leave most of the machine’s bandwidth unused. MLX exists precisely to close that gap.
That is what MLX solves: an open-source array framework (NumPy-like API) purpose-built for Apple Silicon, with first-class support for training and inference. And mlx-lm is its LLM layer: thousands of Hugging Face models with one command, quantization, LoRA and QLoRA fine-tuning, an OpenAI-compatible server, and multi-Mac distributed inference.
2. The Minimum Concepts You Actually Need
Unified memory
CPU and GPU share the same memory pool, so MLX runs ops on either processor without copying weights around. More unified memory means bigger models: roughly 8B on 18GB, 30B+ class models on 64GB+, and 70B on high-memory Macs.
Metal + M5 Neural Accelerators
MLX targets Metal 4 TensorOps and each M5 GPU core carries a dedicated Neural Accelerator for matrix multiplication. Prefill (time to first token) is compute-bound and flies; decode stays memory-bandwidth-bound.
4-bit quantization
Shrinking weights to 4 bits cuts memory roughly 4x with small quality loss. mlx_lm.convert quantizes a 7B model in seconds on a Mac, and every mlx-community *-4bit repo is ready to download and run.
LoRA / QLoRA
Low-Rank Adaptation trains tiny adapter matrices instead of all weights. Point mlx_lm.lora at a quantized model and you get QLoRA: fine-tuning a 7B model on a 32GB Mac at ~250 tokens/sec.
Versions used in this guide
mlx-lm 0.31.3 (April 2026, latest stable on PyPI), Python 3.8+, macOS 15+ recommended for memory wiring on large models. Default model for generate and chat: mlx-community/Llama-3.2-3B-Instruct-4bit.
3. Tutorial: From Zero to Fine-Tuned Model
Everything below runs on any Apple Silicon Mac. No cloud, no Docker, no CUDA. I use the exact commands from the mlx-lm README and LoRA docs, so run mlx_lm.<command> --help whenever you want the full option list.
Step 1 — Install
pip install mlx-lm
pip install "mlx-lm[train]" # needed for LoRA fine-tuningStep 2 — Generate and chat
The default model downloads automatically on first run. Chat keeps context for the whole REPL session.
mlx_lm.generate --model mlx-community/Llama-3.2-3B-Instruct-4bit --prompt "Explain unified memory in one paragraph"
mlx_lm.chat --model mlx-community/Llama-3.2-3B-Instruct-4bitStep 3 — Python API
from mlx_lm import load, generate
model, tokenizer = load("mlx-community/Mistral-7B-Instruct-v0.3-4bit")
messages = [{"role": "user", "content": "Write a haiku about Metal"}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
text = generate(model, tokenizer, prompt=prompt, verbose=True)Step 4 — Convert and 4-bit quantize any HF model
The -q flag quantizes; --upload-repo publishes straight to the mlx-community org. Conversion of a 7B model takes seconds on a Mac.
mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q
mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q --upload-repo mlx-community/my-4bit-mistralStep 5 — Fine-tune with LoRA (or QLoRA)
If --model points to a quantized model you get QLoRA automatically. Data is a folder with train.jsonl plus optional valid.jsonl, or a Hugging Face dataset id. For the full flag list run mlx_lm.lora --help; for repeatable runs use a YAML config with mlx_lm.lora --config config.yaml.
mlx_lm.lora --model mistralai/Mistral-7B-v0.1 --train --data ./my-data --iters 600
mlx_lm.lora --model mlx-community/Mistral-7B-v0.3-4bit --train --data ./my-data --iters 1000 --batch-size 1 --num-layers 4Step 6 — Evaluate, generate, fuse, upload
Test perplexity with --test, generate with the adapter attached, then fuse adapters into the weights and optionally upload or export to GGUF.
mlx_lm.lora --model <path_to_model> --adapter-path ./adapters --data ./my-data --test
mlx_lm.generate --model <path_to_model> --adapter-path ./adapters --prompt "Translate to Spanish: good morning"
mlx_lm.fuse --model mistralai/Mistral-7B-v0.1 --upload-repo mlx-community/my-lora-mistral-7b --hf-path mistralai/Mistral-7B-v0.1Step 7 — Serve it (OpenAI-compatible)
mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit
curl localhost:8080/v1/models -H "Content-Type: application/json"Golden rule
Always run mlx_lm.<command> --help before scripting anything. mlx-lm evolves fast (0.31.3 in April 2026), and the CLI help is the only flag list that cannot be outdated.
4. Benchmarks: M5 vs M4 With MLX (Apple, Sept 2026)
Apple’s Machine Learning Research team published LLM inference numbers on M5 vs M4 MacBook Pro using MLX, with the macOS release that enables M5 Neural Accelerators. The pattern is clean: prompt processing (prefill) is compute-bound and benefits up to ~4x, while token generation (decode) is memory-bandwidth-bound and improves 19–27%.
TTFT dense 14B
<10s on M5
Qwen3-14B 4-bit went from ~36s on M4 to ~8s on M5 in Apple’s test
TTFT 30B MoE
<3s on M5
Mixture-of-experts activates few params per token, ideal for unified memory
Decode (gen) speedup
+19–27% M5 vs M4
Explained by bandwidth: 153 GB/s (M5) vs 120 GB/s (M4), ~28% higher
FLUX-dev 1024px image
3.8x faster on M5
12B diffusion model, same Neural Accelerator story as LLM prefill
Honest read: decode gains are modest because no tensor unit raises the memory ceiling, and independent tests show runtimes like llama.cpp can beat MLX on raw prefill for some models. But for the agentic loop (huge prompts, tool results, whole codebases) the M5’s ~4x prompt-processing jump is the number that matters.
5. Common Errors (and Fixes)
Every one of these bites someone the first week. They are all cheap to fix once you know them.
Model gets killed on load (large model, small Mac)
Check the wired-memory limit with sysctl iogpu.wired_limit_mb and raise it if your Mac has headroom; on macOS 15+ MLX can wire large-model memory. Otherwise drop to a smaller quant (4-bit) or fewer active params (MoE).
Out of memory during LoRA
In order: use a quantized base model (QLoRA), --batch-size 1 plus --grad-accumulation-steps, --num-layers 4 or 8 instead of 16, --grad-checkpoint, shorter sequences. The 32GB recipe in the docs trains Mistral 7B fine.
Adapters do nothing at generation time
You forgot --adapter-path on mlx_lm.generate, or you evaluated a fused model against unfused weights. Generate with the adapter flag; fuse only when you want a standalone model.
Chat answers ignore your instruction format
You skipped the chat template. Always build the prompt with tokenizer.apply_chat_template(messages, add_generation_prompt=True) instead of raw string concatenation.
No Neural Accelerator speedup on M5
You need the macOS release with M5 Neural Accelerator support (MLX targets Metal 4 TensorOps there). Update macOS, update mlx-lm, and re-run; no code changes needed, MLX picks the best kernel per machine.
Stale flags copied from a blog post (possibly this one)
Run mlx_lm.generate --help, mlx_lm.convert --help, mlx_lm.lora --help, mlx_lm.fuse --help. Repos move fast; the CLI is ground truth.
Conclusion
The loop is the whole story: mlx_lm.generate to try, mlx_lm.convert -q to own the weights, mlx_lm.lora to specialize, mlx_lm.fuse to ship, mlx_lm.server to serve. All on a laptop, all on Apple Silicon’s unified memory, and on M5 with Neural Accelerators doing the heavy matrix math.
If you only do one thing this week: pip install mlx-lm and chat with the default 3B model. The moment a local model answers from your own machine, the cloud stops feeling mandatory.
Cheat Sheet
Run
- • mlx_lm.generate --model ... --prompt ...
- • mlx_lm.chat --model ...
- • mlx_lm.server --model ...
Own
- • mlx_lm.convert --model ... -q
- • --upload-repo mlx-community/...
- • load() + generate() in Python
Specialize
- • mlx_lm.lora --train --data ...
- • --test, --adapter-path, --config
- • mlx_lm.fuse [--export-gguf]
