Hosted classifiers like Jev answer from day one but you pay per item forever and you can never touch the weights. Open weights like Laya are free and local but start behind. Anthus built Jev-Flywheel, a small human-in-the-loop feedback layer, and tested both engines on the exact same 140 labels to find out which deal is better.
In this deep-dive I walk you through what the layer does, how to replay the experiment yourself, and the honest numbers: Jev plus layer reaches 0.870, Laya plus layer 0.802, and a full Laya fine-tune jumps to 0.896 — all measured by Anthus on their own corpus, not universal truths.
1. The Problem: a Good Classifier That Never Learns From You
A decision model sorts, scores, or labels your texts and returns a confidence per answer. The moment it disagrees with your team, that disagreement evaporates: the next item gets the same treatment. With Jev you cannot fix this with gradients at all — TypeSafe serves the same weights to everyone. With Laya you can, but full fine-tuning on a few hundred labels brings its own risks: calibration drift and silent changes to every other question on the scorecard.
Jev-Flywheel takes the middle path. The engine stays frozen. A tiny logistic decision head sits on top of its typed answers, and a human approves every adaptation: either refitting the head (changing how much each answer counts) or a steering round where an AI analyst reads the disagreements and proposes a new question. You get a system that learns your conventions without ever retraining the model underneath.
Read the numbers correctly
Every figure in this post comes from one Anthus study on one constructed 8,801-item sentiment corpus with a planted bias (sports skews positive, workplace skews negative). They are Anthus numbers on Anthus data — useful for comparing approaches, not guarantees for your scorecard. Your labels, your templates, your mileage will differ.
2. Minimum Concepts: Head, Refit, Steering Round, Calibration
Four ideas carry the whole design. Learn these and both the tutorial and the results tables will read like plain English.
Decision head
A small logistic model over the engine’s typed answers. It turns answers into one verdict plus a calibrated confidence. Cheap to fit, readable as a few coefficients.
Refit
Re-fitting the head on accumulated labels. Nothing in the engine changes — only how much each answer counts toward the verdict.
Steering round
An AI analyst reads recent disagreements, names the hidden factor in plain English, and proposes a new question (in the recorded run: topic_domain after reading 43 disagreements). A person approves it.
ECE, Brier, auto-accept
ECE measures whether 90% confidence means right 90% of the time. Brier rewards being confident and right. Auto-accept @95% is the production metric: how much traffic clears a 95%-accuracy bar without a human.
The two engines, in one paragraph
Jev (TypeSafe, hosted, jev-1.13.0 in the study) reads the item once and every question rides in the same request, with a 64k-token window. Laya (Convai, Apache 2.0, 421M ModernBERT-large encoder) runs locally but encodes the item once per question (about 8 ms extra per question on an M1 Max), truncates silently past 512 tokens, and is weak on many-option questions (Convai reports 0.425 on Banking77 vs 0.870 for Jev). The flywheel adapter refuses choice questions with more than 20 options.
3. Tutorial: Replay the Paired Comparison Yourself
The repo is deliberately small: one score, a logistic head, no database. These are the exact commands from the Anthus post, verified against the study. You need Apple silicon for the Laya leg (the port is an MLX build); the Jev leg needs a TypeSafe API key.
Step 1
Clone and install
git clone https://github.com/AnthusAI/Jev-Flywheel && cd Jev-Flywheel
make installStep 2
Pull the Laya leg (843 MB download)
make laya # downloads the 843 MB model; about 2 minutes after thatStep 3
Check the paper-600 rows
Replay the paired run: same 140 labels, same refit points, same analyst proposal, same 600 held-out items. The last paper-600 row for each engine should read 0.870 for Jev and 0.802 for Laya. If your numbers match, your harness is faithful to the recording.
What you should see (Anthus figures, 600 held-out items)
| System | Accuracy | ECE | Brier |
|---|---|---|---|
| Jev alone | 0.768 | 0.151 (raw) | 0.188 |
| Jev + flywheel layer | 0.870 | 0.030 | 0.093 |
| Laya alone | 0.722 | 0.107 (raw) | 0.189 |
| Laya + flywheel layer | 0.802 | 0.015 | 0.130 |
| Laya full fine-tune (3 seeds) | 0.896 (0.887–0.903) | 0.087* | 0.089 |
| DistilBERT fine-tune (baseline) | 0.835 (0.818–0.848) | 0.080 | 0.125 |
* Treat the fine-tune’s 0.087 as somewhere between 0.05 and 0.12: the temperature was fitted once per arm on cross-validation stand-ins, and retraining the same seeds moved ECE from 0.116 to 0.046 while accuracy held within a point.
Why this replay design matters
Nothing above the engine adapter knows which engine answered, so the comparison changes exactly one variable. The layer lifted both engines (Jev +10.2 pts, Laya +8.0 pts) and calibrated both — but the 4.7-point gap alone widened to 6.8 points with it. The layer helps both and still leaves the weaker engine behind.
4. Common Mistakes (Most of Them Mine, Some of Them Anthus’s)
The study is unusually honest about what went wrong, including two pre-registered predictions that failed. Steal these lessons instead of re-learning them.
Expecting the weaker engine to gain more
Anthus predicted Laya would gain more from the topic question. It gained +7.2 points vs +10.5 for Jev. Also note the gate promoted refits at different label counts (52 vs 87), so the two steering steps do not start from the same place.
Assuming Laya ships overconfident
Taken from the model card’s raw 0.466 ECE, Anthus predicted Laya’s raw ECE above 0.15. It came in at 0.107 — the port ships Convai’s calibration temperatures. Measure the shipped checkpoint, not the card’s ablation.
Head-only fine-tuning with an aggressive learning rate
Arm B (frozen encoder) scored 0.659, below untuned Laya — but the pre-registered LR grid (1e-4, 1e-3) was too hot, and gentler exploratory rates (1e-5, 2e-5) did not rescue it either. On this task the learning happens in the encoder. Read B as a badly tuned arm, not as proof that freezing fails.
Trusting fine-tuned confidence at face value
The fine-tune wins accuracy (0.896) but its calibration is shaky (ECE ~0.05–0.12, seed-dependent) and its auto-accept @95% swings 59–82% across seeds vs a stable 72% for Jev plus layer. Recalibrate after fine-tuning on held-out data — and 140 labels leaves no room for a calibration split.
Forgetting the drift under the other questions
Fine-tuning Laya on the verdict moved top answers on 8 untrained questions by 9–75% (42% on average; the topic_domain question itself moved 35%). Changed is not wrong — there is no answer key for those — but every other score on that scorecard needs re-validating. A frozen engine never drifts.
Quoting 0.896 as a universal win
Templated corpus, lexical planted cue, 600 held-out items (±1.4 pts), unofficial FP16 port, scripted labeler, no long documents. The fine-tune result is the most exposed to all of this. Anthus says it plainly: a real feedback set’s conventions will not be this easy to read off the words.
5. Which Should You Use?
Anthus’s decision table, condensed. Same study, same caveats — pick by your constraints, not by the biggest number.
0.768 vs 0.722 zero-shot here; Convai’s card says its base checkpoints need fine-tuning on its own benchmark.
Hosted vs yours-to-serve. Laya is free but the serving, batching, and cold starts are your problem.
Laya truncates silently past 512 tokens (Jev lists 64k); Banking77-style questions are a documented Laya miss.
It calibrated both engines (ECE 0.030 / 0.015) and lifted both (+10.2 / +8.0 pts). No drift, readable coefficients.
Free and local, starting several points behind — the layer narrows the cost gap even where it does not close the accuracy gap.
0.896 here at 140 labels — then redo calibration afterwards and re-validate every other question for drift.
The experiment nobody has run yet
Fine-tune Laya only on the observation questions (like topic_domain) and keep the verdict in the readable head. Sharper evidence without giving up the written account — Anthus lists it as future work, and it is the most interesting open door in the whole post.
Conclusion
The flywheel is the safe default: it works with the engine you already pay for, it works with the engine you can host for free, it calibrates both, and it never moves the weights under your other questions. Full fine-tuning is the high card you play when accuracy matters more than explanation — but you pay in calibration work and re-validation.
If you take one habit from this study, make it the pre-registration: Anthus wrote down its predictions, published the two it got wrong, and the paper is stronger for it. Clone the repo, replay the 600 items, and check whether your scorecard behaves like theirs before you bet production on any of these numbers.
Sources
- • Anthus — Jev vs Laya: Same Labels, Same Questions, One Variable (Sep 21, 2026)
- • Anthus — Can You Trust Jev’s Confidence? (Sep 2026)
- • Convai Innovations — Laya model card (Hugging Face, Apache 2.0)
- • AnthusAI/Jev-Flywheel — repo, studies/PREREGISTERED.md
