JevBench v1.6.1 & v1.4.2.2 results, explained
JevBench tracks both JevBench v1.6.1 (Capability, led by Quyet-1.0-Large at 81.7 against Jev 1.13.0’s 77.1 hosted reference and 90.6 Calibration) and JevBench v1.4.2.2 (4-axis harmonic composite, where open 4B models outscore hosted endpoints on unit cost while Jev leads intelligence). Below are both tables, our reproducible evaluation harness, and how System-1 Jev pairs with DeepSeek, Qwen, and o3-mini.
- 77.1Jev 1.13.0 Capability reference on JevBench v1.6.1
- 90.6Jev Calibration on JevBench v1.6.1 · 0.2 s median latency
- 842decisions per system · 534 public + 308 sealed
- 4v1.4.2.2 axes · intelligence, calibration, speed, cost
JevBench v1.6.1 Capability methodology & board
Published 6 October 2026, JevBench v1.6.1 measures text Capability (Intelligence paired with out-of-the-box Calibration) while capping per-decision compute cost. Higher-cost 27B Qwen fine-tunes remain measured as unranked rows alongside TypeSafe’s hosted Jev 1.13.0 reference.
| v1.6.1 Rank | System | Base / Runtime | Capability | Intelligence | Calibration | Est. / 1k | Median | Notes |
|---|---|---|---|---|---|---|---|---|
| Ref | Jev 1.13.0 | TypeSafe System One (hosted) | 77.1 | 63.6 | 90.6 | $0.032 | 0.2 s | Unranked hosted baseline; 64k input context |
| #1 | Quyet-1.0-Large | Gemma-4-31B-it (Apache-2.0) | 81.7 | 73.4 | 90 | $0.045 | 0.4 s | Requires 80 GB GPU; ~8k total prompt budget |
| #2 | deck-31B | Gemma-4-31B-it (self-hosted) | 77.6 | 73 | 82.2 | $0.048 | 0.4 s | Strong intelligence, lower out-of-box calibration |
| #4 | Xor 26B-A4B | Gemma-4-26B-A4B MoE | 72.4 | 58.7 | 86.1 | $0.044 | 0.3 s | MoE active-parameter trade; bf16 deployment |
| #6 | Jev-Omni | Gemma-4-12B (Apache-2.0) | 71.3 | 55.5 | 87 | $0.029 | 0.5 s | Independent open model; not published by TypeSafe |
| #12 | Plumb-4B | Qwen3.5-4B / JevK5 LoRA | 65.4 | 43 | 87.9 | $0.017 | 0.2 s | Low cost per 1k decisions; single GPU self-host |
| #15 | Clef-Flash | Qwen3.5-9B schema head | 63.8 | 42.2 | 85.5 | $0.059 | 0.4 s | Smaller multimodal-trained schema head |
| Capped | Perplexity Decider v1.1 | Qwen3.8-27B noncausal | 82.8 | 75.7 | 89.9 | $0.217 | 0.4 s | Unranked (above v1.6.1 open-weights cost cap) |
| Capped | Kev 27B | Qwen3.8-27B pointer head | 78.9 | 64.2 | 93.7 | $0.529 | 0.5 s | Unranked (above cost cap); serves /v1/systemone |
| Capped | Clef 27B | Qwen3.8-27B schema head | 75.3 | 59.1 | 91.6 | $0.249 | 0.6 s | Unranked (above cost cap); high calibration |
JevBench v1.4.2.2 composite top ten (September September 28, 2026)
Composite first place: Imajev-4B. Highest intelligence: Jev 1.13.0. Best on the hard tier: Plumb-4B. Fastest: Plumb-4B. No one system leads everything — which is why the composite alone is a poor guide.
| Rank | System | Score | Intelligence | Calibration | Speed | Hard tier | Sealed | Runs as |
|---|---|---|---|---|---|---|---|---|
| #1 | Imajev-4B | 67.4 | 52.2 | 80.4 | 90.6 | — | 37% | Self-hosted GPU |
| #2 | Plumb-4B | 65.8 | 53 | 75.5 | 93.5 | 77.7% | 38% | Self-hosted GPU |
| #3 | decider-4b v2 | 64.1 | 49.4 | 75 | 92.9 | 67.3% | 34.7% | Self-hosted, RTX PRO 6000 |
| #4 | Jev 1.13.0 | 63.3 | 53.1 | 76.3 | 83.3 | 74.1% | 36.7% | Hosted API |
| #5 | JevK5 v0.2.0 | 62 | 48.9 | 74.5 | 91.1 | 70% | 33.1% | Self-hosted, RunPod GPU |
| #6 | Cygnet | 61.8 | 49.5 | 74.9 | 90.7 | 75.5% | 33.8% | Self-hosted, RTX PRO 6000 |
| #7 | Hopper | 59.4 | 48 | 79.1 | 86.8 | 65% | 34.1% | Self-hosted, RTX A6000 |
| #8 | Winnow-12B Q8 | 55.6 | 48.3 | 64.8 | 82.3 | 70.9% | 33.1% | Self-hosted, 16 GB GPU |
| #9 | reflex 4B | 54 | 47.5 | 70.4 | 68 | 63.2% | 28.2% | Self-hosted GPU |
| #10 | djev | 52.2 | 47 | 55.4 | 91.4 | 69.5% | 29.9% | Hosted API, preview |
Score, intelligence, calibration and speed are 0–100, higher is better. The hard tier is 220 decisions; sealed is 308 private ones. For Imajev-4B the benchmark publishes the hard tier's two halves — 72.1% public, 75.2% held out — but no combined figure. The fourth axis, cost, is part of the score but not shown; the full table is at the source.
Same numbers, different weighting (v1.4.2.2)
The composite weighs speed and cost as heavily as getting the answer right. JevBench also publishes other views of the same measurements — change the weighting and the top of the board reorders.
| System | Official composite | Accuracy weighted 60% | Intelligence only |
|---|---|---|---|
| Imajev-4B | #1 | #1 | #3 |
| Plumb-4B | #2 | #2 | #2 |
| decider-4b v2 | #3 | #4 | #7 |
| Jev 1.13.0 | #4 | #3 | #1 |
| JevK5 v0.2.0 | #5 | #6 | #8 |
If your pipeline is high-volume and mostly routine, the composite is a fair guide. If the decisions that cost you money are the ambiguous ones, read the accuracy columns instead — that is where Jev's post-training shows up.
Reproducible harness & DeepSeek / Qwen / o3-mini comparison
Use this five-point harness before committing a router, then pair Jev with DeepSeek-R1/V3 or o3-mini only for low-confidence escalations. Copy the runnable Python harness script →
| Check | Sample budget | Measured metric | Reproducibility gate |
|---|---|---|---|
| Accuracy (Blind & Sealed) | 500+ stratified production events (incl. 20–30% holdout) | Exact-match label accuracy & macro-F1 across all classes | Pin prompt wording, question definitions, and class order per run |
| Probability Calibration | 10 confidence deciles (0.0–1.0) over binary / choice outputs | Expected Calibration Error (ECE) & Brier score | Verify 0.85 confidence buckets resolve ~85% true positive rate |
| Tail Latency (p50 / p95) | 200 sequential & concurrent requests from app region | End-to-end TLS + API + parse time (ms) | Exclude warm-up first call; log timeouts & HTTP 429 separately |
| Effective Cost / 1k Decisions | Measured from input token usage header + downstream escalations | Blended $/1k = System-1 input bill + (fallback_rate × System-2 bill) | Include output-token charges when comparing chat models |
| Escalation / Fallback Rate | Swept across confidence thresholds τ ∈ [0.75, 0.95] | % of decisions routed to GPT / DeepSeek / o3-mini or human queue | Report joint accuracy × total blended cost curve per threshold |
| Model family | Architecture | Latency | Cost mechanics | Calibration | Production fit |
|---|---|---|---|---|---|
| TypeSafe Jev (jev-latest / jev-preview) | Single-pass System One model; typed choice / score / noul / box JSON (no generated prose) | ~0.2 s median (v1.6.1) | $0.042 / 1M input tokens ($0.032 / 1k on v1.6.1; $0.0151 / 1k on 350-tok routing), output free | 90.6 / 100 on JevBench v1.6.1 out of the box | High-volume policy gates, LangChain routers, safety checks, triage & batch scoring |
| Qwen System-One Fine-tunes (Plumb-4B / Clef / Kev / Decider) | Open-weight Qwen3.5 / Qwen3.8 fine-tuned with logit or pointer/schema heads | 0.2–0.6 s median self-hosted (v1.6.1) | Self-hosted GPU or $0.017–$0.529 / 1k estimated compute on v1.6.1 | 85.5–93.7 / 100 measured on JevBench v1.6.1 | On-prem / VPC workloads where data cannot leave your GPUs |
| DeepSeek (DeepSeek-V3 / DeepSeek-R1) | Autoregressive chat / CoT token generation (<think> + JSON output via chat/completions) | Variable by reasoning token length (0.8–8+ s on R1 CoT) | Input + generated reasoning/output tokens per request | Raw probabilities uncalibrated without post-hoc logprob temperature fitting | System-2 escalation for ambiguous edge cases, multi-file synthesis, and freeform replies |
| OpenAI o3-mini | System-2 reasoning model with variable thinking budget & structured JSON mode | Multi-second reasoning pass scaling with reasoning_effort | Input + reasoning + visible output tokens per call | Emits discrete choice; requires external verification / self-consistency for confidence | High-stakes fallback path invoked only when Jev confidence stays below threshold |
How to read the score
- Composite — harmonic mean of intelligence, calibration, speed and cost, equally weighted. One weak axis pulls it down hard; a cheap fast 4B can outrank a smarter hosted model.
- Intelligence — how often the system picks the right answer. Sealed decisions make up 20% of this axis.
- Calibration — whether a stated 0.8 really means about 80%. Critical if you set thresholds like “auto-approve above 0.9”.
- Tiers — 72 easy, 96 standard, 146 judge-style, 220 hard public cases. Easy and standard separate almost nothing; the hard tier is where systems disagree.
- Sealed — 308 private cases introduced in v1.4, chance 29.3%. The integrity check on everything above.
Head-to-head pages
Each September comparison breaks this board down to one pair. The October pages — Quyet, deck-31B, Perplexity Decider, Kev, Clef, Xor, Jev-Omni, d1-3B, and Plumb — use JevBench v1.6.1 Capability, which is a different headline from the composite on this page.
FAQ
What changed between JevBench v1.4.2.2 and v1.6.1?
September v1.4.2.2 ranked 91 systems using a four-axis harmonic composite (intelligence, calibration, speed, cost). October v1.6.1 focuses on text Capability — combining Intelligence and Calibration subject to an open-weights cost cap — where Jev 1.13.0 is the hosted reference at 77.1 Capability (63.6 Intelligence, 90.6 Calibration, 0.2 s).
What is JevBench?
JevBench is an independent leaderboard for Jev-style System One decision models, published by Benchmark Heaven. Each system answers the same 842 decisions — 534 public across easy, standard, judge and hard tiers, plus 308 sealed cases whose text stays private.
Is Jev the best model on JevBench?
On v1.6.1 Capability, Quyet-1.0-Large (#1, 81.7) and deck-31B (#2, 77.6) lead Jev 1.13.0 (77.1 reference) on raw intelligence, while Jev keeps higher calibration (90.6 vs 90.0 / 82.2) and 0.2 s median latency. On v1.4.2.2, Jev ranks #4 composite (63.3) and #1 intelligence in the top ten.
Why do small open models rank above Jev on v1.4.2.2?
The v1.4.2.2 composite is a harmonic mean of four equally weighted axes: intelligence, calibration, speed and cost. A self-hosted 4B scores very well on speed and API cost, whereas v1.6.1 isolates Capability (Intelligence + Calibration) with an explicit cost gate.
How does Jev compare to DeepSeek, Qwen, and o3-mini?
Qwen-based System-1 models (Plumb-4B, Clef, Kev, Perplexity Decider) compete directly on JevBench v1.6.1. Autoregressive reasoning models like DeepSeek-R1/V3 and OpenAI o3-mini generate variable-length tokens rather than single-pass calibrated logits, making them complementary System-2 escalation targets behind Jev.