OpenJev published open weights for a decision model that answers typed questions — choice, yes/no probability, or score — over text, web pages, and screenshots, in a single forward pass with up to 52 options.
The headline numbers from its model card: 84.0% against 85.4% for the hosted Jev API on the same 10,000 questions, ~210 ms per web decision on one H100, and quantized FP8 and MLX builds shipped with their own measured accuracy. Here is what changed, what it means for builders, and the exact serving recipe to try this week.
1. Context: a Decision Model, Not a Chatbot
OpenJev does not generate text. You describe the decision in plain words at request time — your own labels, your own options — and it answers with a choice, a yes/no probability, or a score. One model routes a support ticket, flags a policy violation, judges whether an answer is grounded, or tells a browser agent which button to press next.
Under the hood, each of your options gets a letter, and the answer is read from the scores of exactly those letters at the first output position — the server is asked for a single token, never a free-form answer. A calibration step turns those scores into probabilities you can threshold, and the model is tuned to stay consistent when options are reordered: shuffling changes the answer in 2.3% of cases, down from 18.5% before tuning.
Why this matters
No parsing, no chain of thought, no training per task. Labels live in the request JSON, so a new task means changing the JSON, not the model. That is what makes it fast enough to sit inside an agent loop: ~80 ms for a short text decision, ~210 ms for a web step with ~1,460 prompt tokens and ~23 candidate elements.
2. The Numbers That Matter
Every test set was held out from fine-tuning, and the text tables answer the exact same questions on every model. On 10,000 text questions from 34 public sources, OpenJev lands 1.4 points behind the hosted Jev API, 3.7 points above its own base model, and 8.3 points ahead of Nimble 9B.
Agent numbers are strong too: 88.0% next-action accuracy on 2,000 desktop screenshots, 87.4% on unseen websites and 84.5% on unseen domains — and a tie with the hosted API at 39 of 100 end-to-end MiniWoB tasks with the same text-only client. Multilingual holds up: 82.5% on XNLI (from 72.5%) and 85.8% on MASSIVE intent (from 80.4%).
Formats, each with its own measured accuracy
16-bit bf16 · ~54 GB
openjev/openjev
Serve with --quantization fp8 on one 80 GB GPU: the primary measured recipe.
FP8 checkpoint · ~29 GB
openjev/openjev-FP8
84.2% vs 84.0% on the same 10,000 questions (+0.17, CI −0.13 to +0.45); 88.0% vs 88.0% on screenshots. ~2% of answers change.
MLX 8-bit · ~27 GB, text only
openjev/openjev-MLX
Same count as the served model: 8,403 correct (72 questions each way). 1.5% of answers change. No screenshot input.
MLX 4-bit · ~15 GB, text only
openjev/openjev-MLX-4bit
Smallest build for Macs where 27 GB does not fit. 84.3% on the first 4,692 questions. No screenshot input.
What got rejected
An FP4 NVFP4 build (~20 GB) was measured and not published: −1.9 points on text, −0.9 on screenshots. A one-pass readout lives on small score differences, and that build lost too many. A quantized format ships only with its own measured accuracy — a policy worth stealing.
3. Four Implications for Builders
The release is genuinely useful, but three details decide whether you can use it — and one name collision can waste your afternoon.
Open weights ≠ open source
Weights are CC BY-NC 4.0: free for research and non-commercial use with attribution; commercial use needs permission via a discussion on the repo. Only helper/ and serve/ are Apache 2.0. Budget legal review before any production plan.
Keep the calibration settings fixed
Probabilities come from fixed helper settings (READOUT_T=0.85 and friends). Change them and you void every number on the card. Reproduce first, tune later — on your own evals.
Design within the request limits
Up to 52 options per pass (more are grouped and re-passed over winners), prompts to 16,384 tokens, one image per request. Request shapes follow the hosted Jev API, so an existing client can be repointed at your server.
Beware the name twin
Zefan-Cai/Open-Jev (site: open-jev) is a separate project — 2B/9B LoRA adapters on Qwen3.5 with MIT code — not these checkpoints. Same inspiration, different team, different artifacts. Clone the wrong repo and nothing in this guide applies.
Independence note
OpenJev states it is an independent project, not affiliated with TypeSafe; Jev is their product and stays API-only. Treat hosted-vs-local comparisons as the author’s own evals, which is exactly how the card presents them.
4. What to Do This Week
If you have one 80 GB GPU, you can run the primary measured recipe today: vLLM with FP8 quantization plus the helper shim that exposes the decision API. Three commands, all pinned versions from the card.
pip install "vllm==0.29.0" "openai==3.16.2" "httpx==0.28.1"
hf download openjev/openjev --local-dir openjev
# 1. the model
vllm serve ./openjev --host 127.0.0.1 --served-model-name qwen --port 8000 \
--enable-prefix-caching --max-model-len 16384 --gpu-memory-utilization 0.90 \
--limit-mm-per-prompt '{"image":1}' --trust-remote-code --max-num-seqs 256 \
--max-logprobs 64 --gdn-prefill-backend triton --quantization fp8# 2. the decision API in front of it VLLM=http://localhost:8000/v1 TOKENIZER=./openjev \ READOUT_T=0.85 READOUT_NOUL_T=1.829074 READOUT_NOUL_BIAS=0 \ READOUT_TARGETED=1 READOUT_INSTR_STYLE=pyrepr SHIM_STAGGER=1 \ python openjev/helper/shim.py --host 127.0.0.1 --port 3000
curl -s http://localhost:3000/v1/systemone -H 'Content-Type: application/json' -d '{
"model": "openjev",
"state": "Customer message: I was charged twice for my order last week.",
"questions": {
"route": {"type": "choice", "instructions": "Which team should handle this?",
"criteria": {"billing": null, "shipping": null, "technical": null}},
"angry": {"type": "noul", "instructions": "Is the customer angry?"}
}
}'- 1. Serve the model, then the shim, then POST one routing question from your own backlog — compare the choice and its probability against your current classifier.
- 2. Re-run the same question with shuffled option order. If flips exceed low single digits, your labels — not the model — are ambiguous.
- 3. On a Mac with no NVIDIA GPU, try the MLX 8-bit build for text-only triage; keep screenshots on the server build.
- 4. Before anything commercial, open the licensing discussion on the repo — CC BY-NC 4.0 blocks production use without permission.
Agent-builder tip, straight from the card
Treat DONE as the model’s opinion and confirm completion in your own loop — a success message, a changed URL, a saved record — before you stop. Typed decisions reduce parsing bugs; they do not remove verification.
Conclusion
OpenJev is the rare open-weights release that ships with honest deltas: 1.4 points behind hosted on text, tied on MiniWoB, quantized builds measured rather than assumed, and a rejected FP4 disclosed instead of buried.
The catch is the license — research-friendly, production-gated — and the discipline it demands: fixed calibration, 52-option passes, verified completion. Run the recipe this week on your own questions; the card gives you everything except your labels.
