Mistral Large 4 Open Weights: Self-Hosting Guide, GPUs and License

On October 29, 2026, Mistral published the weights of Mistral Large 4, the ~1T-parameter Mixture-of-Experts model it had been serving as an API-only preview since October 6. Two official checkpoints landed on Hugging Face: FP8 at about 1.06 TB and NVFP4 for Blackwell GPUs at about 610 GB, under a new Mistral Ecosystem Open-Weight License. Downloading them is free. Running them takes a datacenter node.

This is the follow-up to my preview post: who can realistically serve it, the exact vLLM and SGLang commands, what the license allows and when the API is the better deal. On November 3, 2026, I read both model cards, the LICENSE file, config.json, Mistral’s technical report, the vLLM recipe and the SGLang cookbook, and redid the parameter, memory and KV-cache arithmetic myself. I didn’t download the weights or run the model, so there are no first-hand speed claims here.

1. The spec card

Mistral nicknames it “Le Chonk”. Everything below comes from the model card, config.json, Mistral’s docs and the Hugging Face API, checked on November 3, 2026.

Mistral Large 4 at a glance

Parameter counts are Mistral’s. I re-derived them from config.json and got ~1.05T total, 49.8B active per token and 52.3B with the embedding and output layers.

Spec	Value
Architecture	Granular MoE: 72 layers, 128 routed experts (4 per token) plus 1 shared expert; the first layer is dense
Parameters	~1T total (1.05T in Mistral’s docs); 49B active per token, 52B with embeddings and output layers; 1.6B Pixtral vision encoder
Attention	Multi-head latent attention (MLA) with a DeepSeek-style sparse indexer that picks 2,048 keys per query; 9 of the 72 layers reuse another layer’s KV cache
Context	1,048,576 tokens in config.json; vLLM validated up to 256K
Input → output	Text and images → text; function calling, strict JSON schema output, 160+ languages per the card
Reasoning	reasoning_effort low, medium or high (the default); it can’t be turned off
Speculative decoding	Built-in DSpark draft head (3 BF16 MLA layers proposing 3-token blocks); Mistral measured up to 1.9× throughput
Official checkpoints	FP8 block-scaled (1,061 GB) and NVFP4 experts with FP8 linears (610 GB), per the Hugging Face API
Community formats	GGUF from ggml-org and Unsloth (233 GB to 2.1 TB); AMD Quark MXFP4 (578 GB)
License	Mistral Ecosystem Open-Weight License, a modified MIT (details in section 5)
API	mistral-large-4 (also listed as mistral-large-4-0): $0.70 input, $0.07 cached input, $2.10 output per 1M tokens

Release timeline

• October 6, 2026: public preview on the Mistral API and Mistral Studio. The Hugging Face repo was an empty placeholder.
• October 29, 2026, 14:11 UTC: “Initial release” commits on both official repos, the same day as Hugging Face’s launch post and Mistral’s announcement.
• Around October 28–29: GGUF conversions from ggml-org and Unsloth, plus Docker images tagged mistral-large-4 for vLLM and SGLang.
• November 2: Mistral adds a 30-page technical report to the FP8 repo.
• November 3: Mistral’s docs still label the API model “Public Preview” (version v26.10).

49B or 52B active? Both, from the same card

The repo name says A52B, Hugging Face’s blog says 49B, and coverage picks one or the other. The model card gives both: 49 billion active parameters per token, 52 billion counting embeddings and output layers. My arithmetic from config.json agrees: 4 routed experts plus the shared expert, attention and the dense first layer come to about 49.8B, and the 204,800-token embedding and output matrices add 2.5B. For memory planning, though, the number that matters is the total, because every expert has to sit in GPU memory.

2. What changed since the preview

My preview post covered the API-only phase in mid-October. Four things changed with the release. Mistral’s announcement calls the published checkpoint an “improved” Mistral Large 4, with more reinforcement learning on coding and cybersecurity and better reasoning efficiency.

The four changes that matter

📦

Weights and a named license

FP8 and NVFP4 checkpoints, plus a license text that didn’t exist in October. Self-hosting is now a hardware question, not a promise.

🎚️

Three reasoning levels

low, medium and high (the default). Mistral says low replaces the preview’s “none”, and Hugging Face notes reasoning can’t be disabled anymore.

💶

About half the list price

The API now lists $0.70 / $2.10 per 1M tokens (cached input $0.07), against the preview’s $1.36 / $4.18 list and $0.68 / $2.09 launch sale.

⚡

A drafter inside the checkpoint

The DSpark speculative-decoding head ships with both checkpoints, and vLLM and SGLang turn it on with one flag.

Score changes, preview → release (vendor-reported)

• Terminal-Bench 4.0: 28.3% → 41%, with 30% fewer output tokens, per Mistral’s announcement.
• DeepSWE 1.1: 61.7% at the preview → 65.2% in the technical report.
• Lakera B3 agent security: 93.3% → 97.1% attack resistance.
• Artificial Analysis Coding Agent Index: 48% on the card (48.3% in the report), “effectively tied” with GLM-5.3 at 48.9%. That’s a close second on Mistral’s own table, not first. Hugging Face’s post still quotes the preview’s 49.8%, but the report labels the index v2, so don’t compare the two.

If you read the preview post

Its advice (evaluate through the API, keep Large 3 if you must self-host) made sense while nothing was downloadable. The trade-off is different now: Large 4 runs on a single Blackwell node, but under a license with a Model-as-a-Service clause that Large 3’s Apache 2.0 never had.

3. Hardware math: who can actually run it

Start with the fact that decides everything. A Mixture-of-Experts model computes only about 49B parameters per token, but all ~1.05T must sit in GPU memory, because the router can send any token to any expert. Routed experts are 98% of the parameters (1.029T), which is why quantizing only the experts shrinks the file so much.

Weights: bytes per parameter × parameters (my arithmetic)

• FP8 is 1 byte per weight. 1.029T expert weights ≈ 1,029 GB, plus attention, embeddings, the vision encoder and the drafter, some of them kept in BF16: 1,061 GB on Hugging Face.
• NVFP4 is 4 bits per weight plus one FP8 scale per 16 weights: 4.5 bits, or 0.5625 bytes. 1.029T × 0.5625 ≈ 579 GB of experts, plus ~30 GB of FP8 and BF16 layers: 610 GB on Hugging Face (606 GB in the technical report).
• That’s why the preview-era rule of thumb, about 525 GB at 4 bits, was too optimistic: the scales and the layers left unquantized add roughly 85 GB.

Then the KV cache. For each token, every one of the 63 layers that keep their own cache stores a compressed 576-value latent (512 values plus a 64-value RoPE key). In BF16 that’s 63 × 576 × 2 bytes ≈ 72.6 KB per token. The default FP8 cache roughly halves it to about 40 KB, plus a small cache for the sparse indexer. A full 1,048,576-token conversation therefore needs about 76 GB in BF16, or about 42 GB in FP8.

The catch: with plain tensor parallelism, every GPU keeps its own copy of that compressed cache. Mistral’s technical report says so directly (“naive tensor parallelism… suffers from KV replication across TP ranks”), and it’s why the vLLM recipe’s decode-context-parallel layout (DCP8) holds 28.76M KV tokens on 8× GB300 instead of 3.60M. So the memory that matters is what’s left on each GPU, not the node total.

Will it fit? Per-GPU budget at --gpu-memory-utilization 0.85

My arithmetic: checkpoint size ÷ GPU count, against 85% of each GPU’s memory (H200 141 GB, B200 180 GB, B300 and GB300 288 GB). The recipe’s own per-GPU figures (~128 GiB FP8, ~76 GiB NVFP4 on 8 GPUs) run a bit higher, because some pieces are replicated. “Left” still has to hold activations, CUDA graphs and the KV cache. Status comes from the vLLM recipe and the SGLang cookbook.

Setup	Checkpoint	Weights per GPU	Left per GPU	Status
8× H200	FP8 (1,061 GB)	132.6 GB	None (budget 119.9 GB)	The vLLM recipe says FP8 needs two H200 nodes; Hugging Face’s Endpoints recipe claims 8× H200 at 131K context
2 × 8× H200	FP8	66.3 GB	~54 GB	Validated with TP16 + DCP8 (or PP2 × TP8); no speed numbers published
4× B200	NVFP4 (610 GB)	152.5 GB	~0.5 GB	Doesn’t fit
8× B200	NVFP4	76.3 GB	~77 GB	Validated (vLLM)
8× B200	FP8	132.6 GB	~20 GB	Not on the validated list; little room left for KV cache
4× B300 / GB300	NVFP4	152.5 GB	~92 GB	Validated: the vLLM default and SGLang’s TP4 example
8× B300	FP8	132.6 GB	~112 GB	Validated; SGLang measured a 2.49M-token FP8 KV pool
8× AMD MI350X	FP8, experts re-quantized to MXFP4 at load (~578 GB)	—	—	Validated: two TP4 replicas per node with vLLM’s ROCm image

What the KV numbers mean in practice

• 8× B300 with FP8: SGLang’s measured 2.49M-token pool is about two full 1M-token conversations, or 19 at 128K. With a BF16 cache it drops to 1.34M.
• Long agent sessions run out of KV cache before they run out of compute. If yours do, look at the recipe’s DCP presets before you buy more GPUs.

Smaller setups: GGUF, with patience

Hugging Face’s launch post says the 2-bit GGUFs (Q2_K at 381 GB, IQ2_XXS at 298 GB) fit on one Apple Silicon Ultra with 512 GB of unified memory. On an M3 Ultra with Q2_K, the HF team measured about 90 tokens/s of prefill and 15 tokens/s of generation. On 4× RTX 6000 Pro they got ~10 tokens/s with the NVFP4 GGUF at 32K context, and 3 to 30 tokens/s with IQ2_XXS, depending on context length. Fine for private evals, not for production.

4. Quickstart: vLLM and SGLang

Both servers need model-specific Docker images for now. The vLLM recipe says no vLLM release or nightly wheel includes Mistral Large 4 support yet, so you run vllm/vllm-openai:mistral-large-4; SGLang uses lmsysorg/sglang:mistral-large-4. Every serving command below is copied from the recipe or a model card; I only added comments. If you’re still choosing between the two engines, my vLLM vs SGLang guide shows how to benchmark both on your own traffic.

Before you start

From the vLLM recipe, the SGLang cookbook, the model cards and Docker Hub, checked November 3, 2026.

Piece	What to use	Notes
vLLM	vllm/vllm-openai:mistral-large-4 (the recipe lists vLLM ≥ 0.34.0)	Mistral’s recommended server; vllm/vllm-openai-rocm:mistral-large-4 on AMD
SGLang	lmsysorg/sglang:mistral-large-4	The cookbook covers both checkpoints and adds --tokenizer-mode mistral as an option
Download	1,061 GB (FP8) or 610 GB (NVFP4)	At 1 GB/s the FP8 repo takes about 18 minutes; at 100 MB/s, about 3 hours

vLLM: NVFP4 on 4× GB300 (the recipe’s default)

Blackwell only. The timeout gives the engine up to 100 minutes to load. On 8× B200, the recipe’s command is identical except for --tensor-parallel-size 8.

# vLLM recipe default: 4× GB300, NVFP4, tensor parallel 4
export VLLM_ENGINE_READY_TIMEOUT_S=6000
vllm serve mistralai/Mistral-Large-4-1T-A52B-NVFP4 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.85 \
  --tensor-parallel-size 4 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}'

vLLM in Docker: NVFP4 on 8× B200

The recipe’s Docker form. For FP8 on 8× B300, the recipe’s command is the same with mistralai/Mistral-Large-4-1T-A52B as the model.

# Same recipe, Docker form, on 8× B200 (NVFP4, tensor parallel 8)
docker run --gpus all \
  --privileged --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e VLLM_ENGINE_READY_TIMEOUT_S=6000 \
  vllm/vllm-openai:mistral-large-4 mistralai/Mistral-Large-4-1T-A52B-NVFP4 \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.85 \
  --tensor-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}'

vLLM: FP8 on two 8× H200 nodes (TP16 + DCP8)

The recipe’s long-context H200 preset. Set $HEAD_IP and $IFACE_NAME (the network interface) on both nodes, and send requests to node 0. Correctness was validated here, but no speed numbers were taken.

# Head (node 0)
export VLLM_ENGINE_READY_TIMEOUT_S=6000
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
vllm serve mistralai/Mistral-Large-4-1T-A52B \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.85 \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr $HEAD_IP \
  --decode-context-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}'

# Node 1
export VLLM_ENGINE_READY_TIMEOUT_S=6000
export GLOO_SOCKET_IFNAME=$IFACE_NAME
export NCCL_SOCKET_IFNAME=$IFACE_NAME
vllm serve mistralai/Mistral-Large-4-1T-A52B \
  --tokenizer-mode mistral \
  --config-format mistral \
  --load-format mistral \
  --max-model-len 262144 \
  --max-num-batched-tokens 8192 \
  --max-num-seqs 128 \
  --gpu-memory-utilization 0.85 \
  --tensor-parallel-size 16 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr $HEAD_IP \
  --headless \
  --decode-context-parallel-size 8 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral \
  --reasoning-parser mistral \
  --speculative-config '{"method":"dspark","num_speculative_tokens":3}'

SGLang: NVFP4 on 4 Blackwell GPUs (NVFP4 model card)

Run it inside the image. DSpark is on, and --max-running-requests is pinned to 64 (SGLang falls back to 48 without it). For FP8, the main card uses mistralai/Mistral-Large-4-1T-A52B with --tp-size 8.

docker pull lmsysorg/sglang:mistral-large-4

sglang serve \
  --model-path mistralai/Mistral-Large-4-1T-A52B-NVFP4 \
  --tp-size 4 \
  --mem-fraction-static 0.85 \
  --speculative-algorithm DSPARK \
  --max-running-requests 64 \
  --reasoning-parser auto \
  --tool-call-parser auto \
  --host 0.0.0.0 \
  --port 30000

Call it with the OpenAI client (adapted from the model card)

max_completion_tokens counts reasoning tokens too. The reasoning field comes from the card’s vLLM example; with SGLang, print the raw message once, since field names can differ between servers.

# pip install openai
from openai import OpenAI

# vLLM listens on port 8000; the SGLang command above uses 30000
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
model = client.models.list().data[0].id

response = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Find the race condition in this handler: ..."}],
    reasoning_effort="medium",   # "low" | "medium" | "high" (default)
    temperature=1.0,
    top_p=0.95,
    max_completion_tokens=4096,  # counts reasoning tokens too
)

print(response.choices[0].message.reasoning)  # reasoning trace (vLLM)
print(response.choices[0].message.content)    # the answer

Mac Studio with 512 GB: a 2-bit GGUF (Hugging Face’s post)

Uses the Llama App CLI from llama.app. The post’s examples then query the usual /v1/chat/completions endpoint.

# Llama App CLI (llama.app); Q2_K is a ~381 GB download
llama serve -hf ggml-org/Mistral-Large-4-1T-A52B-GGUF:Q2_K

Flags worth understanding

• --tool-call-parser mistral and --reasoning-parser mistral (vLLM), or auto (SGLang): required for tool calls and to split the reasoning from the answer.
• --tokenizer-mode, --config-format and --load-format mistral: in the recipe, missing from the model card’s simpler Docker command. The recipe says its commands take precedence, so keep them.
• --speculative-config with dspark (vLLM) or --speculative-algorithm DSPARK (SGLang): turns on the built-in drafter. The checkpoint fixes the depth at 3.
• --kv-cache-dtype fp8_ds_mla: the NVFP4 card sets it explicitly, while the recipe says the FP8 latent cache is already the default. Either way you get the FP8 cache.
• --decode-context-parallel-size 8: splits the KV cache across ranks for long context instead of copying it to every GPU.

Settings from the model card

temperature 1.0 and top_p 0.95 at every reasoning level. Use reasoning_effort high for agents and hard prompts, medium for day-to-day work and low for quick chat. Keep the toolset small and well defined, and write a system prompt that describes the environment.

5. Honest limits

Published weights don’t make this an easy model to run. These are the limits I’d put in front of a team.

Where I’d be careful

• It needs a Blackwell node or two Hopper nodes. FP8 doesn’t fit a single 8× H200 server at the recipe’s settings, and NVFP4 needs Blackwell.
• Support is days old. No vLLM release includes it yet, and both servers run from model-specific images. Expect changes as support moves upstream.
• Most benchmarks are Mistral’s, and independent harnesses can disagree. Vals measured 26.26% on Terminal-Bench 4.0 against Mistral’s 41%, running reasoning on high through Mistral’s API.
• On aggregate intelligence it trails the best open models: Artificial Analysis gives it 26 on its Intelligence Index, against 32 for DeepSeek V4.1 Flash and GLM-5.3. Mistral’s pitch is coding, cybersecurity and token efficiency.
• The 1M context is advertised, but vLLM validated 256K (its --max-model-len 262144), and long agent sessions hit the per-GPU KV limits from section 3.
• Reasoning can’t be turned off, and high effort is verbose. Measure tokens per task, not just tokens per second.
• Mistral says it is “modifying our guardrails” after seeing “a potential for uplift to conduct real attacks”. Those guardrails live in its hosted service; a self-hosted copy has only what’s trained into the weights, so filtering is your job.
• Mistral hedges too: both model cards recommend its API “if local serving is subpar”.

The license in plain words

The Mistral Ecosystem Open-Weight License is the MIT text with three numbered conditions. You may use, copy, modify, fine-tune, distribute and sell copies of the model and its derivatives, as long as you keep the copyright and permission notice. There’s no revenue threshold, no acceptable-use list and no clause against training competing models; the model card adds only that you must not infringe third parties’ rights. The catch is Model as a Service. If you or any affiliate give third parties inference or post-training access to the model or a derivative “in a manner that allows such third party to exercise meaningful control over the inputs, parameters, or training data” (an API, a hosted endpoint, fine-tuning as a service), you need a separate commercial license from Mistral before using it for any commercial purpose. Exempt: internal use, end-user products where the model is embedded in specific features, relaying requests to models hosted by others, and use through Mistral’s own products. A SaaS feature or an internal coding agent is fine; reselling Large 4 tokens isn’t, without a deal. I’m not a lawyer, so have counsel read the one-page text before you ship.

Verdict: self-host it, or call the API?

Mistral delivered what it promised in October: real weights, two practical checkpoints, day-one vLLM and SGLang images and a license that’s permissive for almost everyone except inference providers. A trillion-parameter model trained in Europe now runs on a single Blackwell node you control.

For most teams, though, the API is still the better deal. At $0.70 / $2.10 per 1M tokens, you need a busy node, long uncached prompts or a hard data-residency rule before owning the hardware pays off. Self-host when control is the requirement, not when you’re hoping to save money.

Break-even sketch (my arithmetic, an illustration only)

Take vLLM’s best measured B200 layout (8 prefill + 8 decode GPUs, NVFP4, 32K-token inputs): 164.4 output tokens/s per GPU, or 591,840 per GPU-hour. At the API’s $2.10 per 1M output tokens, that’s $1.24 an hour. Now assume 1,000 output tokens per request (my assumption; the recipe doesn’t say): each GPU-hour also processes about 19M input tokens, worth $13.26 uncached at $0.70/M, or $1.33 cached at $0.07/M. Lambda lists an HGX B200 at $6.69 per GPU-hour on demand. So a fully loaded node with uncached prompts produces ~$14.50 of API-priced tokens per hour and pays off if it’s busy more than ~46% of the time. With cache-friendly prompts it’s ~$2.57, and the API wins. Prefix caching speeds up your own server too, so treat that second case as a rough bound, and plug in your own token mix.

Who should do what

4× B300/GB300 or 8× B200

Teams that already run Blackwell, need data to stay in-house or want to fine-tune. Start with NVFP4 and the recipe’s default command.

Two 8× H200 nodes

Workable with FP8 and TP16 + DCP8, but nobody has published speed numbers. Load-test it before you commit traffic.

Inference providers and GPU clouds

Offering Large 4 as an API or hosted endpoint is Model as a Service: get Mistral’s commercial license first.

Everyone else

Use the API: $0.70 / $2.10 per 1M tokens, no hardware, and Mistral’s hosted guardrails included.

Self-hosting checklist

1. Run your own prompts through the API first at low, medium and high effort, and log quality and tokens per task. That’s your baseline.
2. Pick the checkpoint by GPU: NVFP4 on Blackwell, FP8 on Hopper or AMD.
3. Check the per-GPU math in section 3 against your longest real context, not the 1M headline.
4. Pull the model-specific Docker image and start from the recipe’s command, not an old vLLM install.
5. Re-run the API baseline against your server, with and without DSpark, then redo the break-even math with your real utilization.
6. If anyone outside your company will call the model, read the Model-as-a-Service clause with counsel.