Every agent pipeline I ship has the same boring step: force an LLM to pick one option, then pray the JSON parses. Open-Jev attacks exactly that step. It is an independent, open research project by Zefan Cai that turns Qwen checkpoints into typed decision models: you hand it a context plus explicit questions, and it returns probabilities over your choices directly — no autoregressive text, no JSON to parse.
I read the project page, the evaluation reports, and both Hugging Face cards so you don’t have to. Below: the spec sheet for the 2B/9B adapters plus the 0.4B DeBERTa sibling, what genuinely improves over prompting, a working quickstart, the honest limits — including the fact that the 27B run is still pending — and my verdict.
1. Spec Sheet: Two Families, One Interface
There are two artifacts to keep apart. The main line is Open-Jev: LoRA adapters plus a scalar decision head trained jointly on Qwen3.5 2B and 9B. The sibling is com-kotobalabs/open-jev-deberta-v3-large: a 0.4B DeBERTa-v3-large encoder with a 3-layer scoring head. Both expose the same three typed questions — choice, noul (yes/no probability), and score (ordered rubric) — and both answer in one forward pass.
Open-Jev 2B
LoRA adapter + decision head on Qwen3.5-2B. Hard-label accuracy 94.71% test / 86.02% OOD, audited.
Open-Jev 9B
LoRA adapter + decision head on Qwen3.5-9B. Hard-label accuracy 97.54% test / 91.97% OOD, audited.
Open-Jev 27B
Qwen3.8-27B run has started optimizer updates on 148,639 frozen rows. Final results pending — no accuracy to quote yet.
DeBERTa variant
0.4B params, choice over up to 255 options, score over 2–10 levels, noul. 512 tokens total (state cut to 256). Apache-2.0.
License
Original Open-Jev code is MIT. Model packages require pinned upstream Qwen weights; the dataset card documents redistribution exclusions with a reconstruction guide.
Status
Research preview, September 2026. 2B/9B checkpoints released; 27B training in progress; DeBERTa variant published on Hugging Face.
27B is pending — stated explicitly
The project page is unambiguous: a new 27B iteration on 148,639 frozen training rows has started optimizer updates and its final results are pending. Every accuracy number in this post belongs to the released 2B/9B checkpoints. Treat any 27B claim you see elsewhere as speculation until the audited table updates.
Held-out scale behind the headline numbers
Each completed model is measured on 26,452 held-out decision rows, of which 25,492 hard-label rows feed the accuracy (10,046 test + 15,446 OOD). That is a serious held-out, not a 200-question demo split — with the caveat that the labels are synthetic references, covered in the limits section.
2. What Improves Versus Prompting an LLM
The pitch is not “smarter model” — it is “cheaper, typed primitive”. If your pipeline already forces a big model to output a label, Open-Jev replaces that call with a small model that only ever returns distributions. The numbers below are project-reported and independently audited per the evaluation report — I flag exactly what they do and don’t cover.
9B hard-label: 97.54% test / 91.97% OOD
2B at 94.71% / 86.02%→ 97.54% / 91.97%
The 9B adapter gains ~3 points in-domain and ~6 points out-of-distribution over the 2B on 25,492 hard-label rows. OOD is the number that matters for real routing, and 92% there is genuinely strong.
Zero structured-output failures, by construction
LLM + JSON schema + retries→ One forward pass → distribution
Nothing is generated, so there is nothing to parse. The DeBERTa card puts it bluntly: the structured-output error rate is 0 by construction. Every flaky parse-and-retry loop in your agent harness disappears.
Up to 255 options in one pass (DeBERTa)
52-option batches on the Qwen line→ 255 options, single pass
The 0.4B variant scores every option in one forward pass — banking77 intent over 77 options hits 0.916 accuracy. For intent routing with big label sets, that is the whole game.
Calibrated probabilities, not vibes
Verbalized confidence→ ECE 0.022 in-domain
Post-hoc temperature fitted on a validation split gives in-domain expected calibration error of 0.022 (DeBERTa) and a saved temperature of ~1.897 on the 9B. You can actually threshold these numbers.
CPU-viable tier exists
H100-class serving→ 0.4B on CPU
The DeBERTa variant runs fp32 on an M1 Max CPU (1.8 s for 4 questions) and 28 ms end-to-end for 10 questions on one H100, 518 questions/s at batch 8. Routing no longer needs a GPU per call.
Training recipe is public and tiny
Opaque distillation→ LoRA r8 + head, public data
9B: 20,204 optimizer steps at batch 4, LoRA rank 8 / alpha 16, head initialized from the Yes-minus-No readout. DeBERTa: 18,000 states / 42,000 questions, 1 epoch, ~229 s on one H100. You can reproduce the shape of this.
Where JevBench fits
Separately, the project ran the public JevBench subset — 231 of 534 tasks, the rest private — over the released 2B/9B checkpoints. Candidate order differs on 119 of 139 choice tasks and timing is diagnostic, not a hardware-normalized speedup. Useful context, not a headline; the audited full-data table above is the number to quote.
3. Quickstart: Decisions in Minutes
Two verified paths. The DeBERTa variant is the fastest thing you can run today — its snippet below comes straight from the model card. The Qwen adapters need their pinned upstream base weights plus the Open-Jev loader, so I give you the exact pointers instead of invented commands.
Option A — DeBERTa 0.4B (pip + decide, from the card)
# pip install torch transformers safetensors huggingface_hub sentencepiece protobuf
import sys; sys.path.insert(0, “<path to this repo snapshot>”)
from typed_decisions.open_jev import OpenJev
m = OpenJev.from_pretrained(“com-kotobalabs/open-jev-deberta-v3-large”)
m.decide(
“I was charged twice for the same order. I want my money back now.”,
[{“type”: “choice”, “instructions”: “Which product area is the message about?”,
“options”: [“fees & charges”, “refund & dispute”, “card”, “other”]},
{“type”: “noul”, “instructions”: “The customer is asking for a refund.”}])
# → [{’choice’: ..., ’probabilities’: {...}, ’confidence’: ...}, {’noul’: ...}]Option B — Open-Jev Qwen adapters (repo + checkpoints)
git clone https://github.com/Zefan-Cai/Open-Jev # checkpoints: https://huggingface.co/collections/ZefanCai/open-jev # ZefanCai/Open-Jev-2B and ZefanCai/Open-Jev-9B # data splits + manifests: https://huggingface.co/datasets/ZefanCai/Open-Jev # NOTE: adapters require the exact pinned upstream Qwen revision # stated in each package (e.g. 9B → c202236235762e1c871ad0ccb60c8ee5ba337b9a)
Option C — Sanity check before you trust it
# 1. Re-run the 76-case provider comparison inputs on your own labels. # 2. Re-calibrate temperature on YOUR data (OOD is over-confident by ~0.03). # 3. Shuffle option order twice — the models are trained for order # consistency, but your harness should verify it per deployment.
Serving notes that matter
A generic AutoPeftModel text-generation call does not implement the decision interface — you must use the Open-Jev loader so the separate decision head and saved temperature apply. Prefix caching is opt-in and disabled in the reported measurements; keep it off until you have reproduced the baseline.
Tip: start with the 0.4B, graduate to the 9B
Prototype routing, intent classification, and guardrails on the DeBERTa variant — it runs on CPU and its limits are documented per subgroup. Move to the 9B adapter only when OOD accuracy on your own labels justifies a GPU in the loop.
4. Honest Limits
Decision models look like magic until you read the subgroup tables. Here is what I would check before putting any of these in a production path.
Synthetic reference labels, not task win rates
The 94–97% figures measure agreement with synthetic reference labels, not gameplay or workflow completion rates. A later coverage audit found equivalent game actions and omitted policy details. High label agreement is not proof your workflow completes.
Aggregates hide weak subgroups
9B Wiki OOD expected accuracy is 32.74%, reasoning OOD hard-label accuracy is 73.60%, and the T-Rex test slice has only 4 rows. If your workload looks like the weak slices instead of the average, the headline number does not apply to you.
The 27B does not exist yet as a result
Optimizer updates started on 148,639 frozen rows, with a further 96,849-row community stage queued behind it. The page states plainly that neither preparation nor training progress establishes an accuracy gain. Do not plan capacity around 27B numbers.
DeBERTa: English-only, 512 tokens, three domains
Banking support, movie reviews, Wikipedia yes/no — anything else is out of distribution and must be measured first. Unseen ordered scales are weakest (0.45 on new level sets vs 0.26 majority). State is cut to 256 tokens of the 512 total.
Data comes with redistribution exclusions
The dataset card documents redistribution exclusions and a reconstruction guide for the original training mixtures instead of a plain download. Code is MIT, but data and upstream Qwen weights keep their own licenses — read both before commercial use.
What I would not do
I would not quote the 97.54% without the OOD number next to it, I would not deploy on the 27B’s behalf before audited results land, and I would not skip re-calibration on my own data — OOD over-confidence of ~0.03 mean is small until it sits in front of a refund decision.
5. Verdict: Who Should Adopt It
If your agent harness forces big models to emit labels, Open-Jev is the most principled cost cut available in the open right now: a 9B adapter at ~92% OOD agreement with calibrated outputs, and a 0.4B CPU variant that deletes the parse-retry loop entirely. Start with the DeBERTa card snippet on your own labels today.
If you need generation, arguments, or long context, this is not your model — it chooses among options you give it, nothing more. And if your decision surface is multilingual or far from the three training domains, measure OOD first: the 0.854 → 0.690 in-domain-to-OOD gap on the small variant is the honest preview of what awaits.
My call
Adopt-and-measure for routing, guardrails, and agent verifiers — 9B where a GPU is affordable, 0.4B everywhere else. Watch the 27B run, but don’t plan on it. This is the rare open release whose limits section is as useful as its accuracy table.
Conclusion
Open-Jev reframes the small-model story: instead of a worse generator, you get a better decider. Direct probabilities, a three-word typed interface, audited 2B/9B numbers, and a CPU-tier sibling make it the cleanest open answer to TypeSafe’s Jev shape I have seen — with the 27B still an open question, as the authors themselves state.
Every figure above traces to the two sources below, with the pending and weak spots flagged, not hidden. If you benchmark it on your own routing labels, I want to hear your OOD delta.
Sources
Cheat sheet
- • LoRA + decision head on Qwen3.5 2B (94.71%/86.02%) and 9B (97.54%/91.97%), audited on 26,452 held-out rows
- • 27B on Qwen3.8: training started, results pending — no numbers exist yet
- • 0.4B DeBERTa sibling: 255 options, CPU-viable, 0.854 in-domain / 0.690 OOD
