jevmodel.org account

Sign in for 100,000 free input tokens

About 500 Jev requests in the playground or from your own code with an API key. Packs start at $9.90 when you need more. No card required.

By signing in you agree to our terms. We only use your email for your account.
Leaderboard guide · JevBench v1.6.1 (6 October 2026) & JevBench v1.4.2.2

JevBench v1.6.1 & v1.4.2.2 results, explained

JevBench tracks both JevBench v1.6.1 (Capability, led by Quyet-1.0-Large at 81.7 against Jev 1.13.0’s 77.1 hosted reference and 90.6 Calibration) and JevBench v1.4.2.2 (4-axis harmonic composite, where open 4B models outscore hosted endpoints on unit cost while Jev leads intelligence). Below are both tables, our reproducible evaluation harness, and how System-1 Jev pairs with DeepSeek, Qwen, and o3-mini.

01

JevBench v1.6.1 Capability methodology & board

Published 6 October 2026, JevBench v1.6.1 measures text Capability (Intelligence paired with out-of-the-box Calibration) while capping per-decision compute cost. Higher-cost 27B Qwen fine-tunes remain measured as unranked rows alongside TypeSafe’s hosted Jev 1.13.0 reference.

v1.6.1 RankSystemBase / RuntimeCapabilityIntelligenceCalibrationEst. / 1kMedianNotes
Ref Jev 1.13.0 TypeSafe System One (hosted) 77.1 63.6 90.6 $0.032 0.2 s Unranked hosted baseline; 64k input context
#1 Quyet-1.0-Large Gemma-4-31B-it (Apache-2.0) 81.7 73.4 90 $0.045 0.4 s Requires 80 GB GPU; ~8k total prompt budget
#2 deck-31B Gemma-4-31B-it (self-hosted) 77.6 73 82.2 $0.048 0.4 s Strong intelligence, lower out-of-box calibration
#4 Xor 26B-A4B Gemma-4-26B-A4B MoE 72.4 58.7 86.1 $0.044 0.3 s MoE active-parameter trade; bf16 deployment
#6 Jev-Omni Gemma-4-12B (Apache-2.0) 71.3 55.5 87 $0.029 0.5 s Independent open model; not published by TypeSafe
#12 Plumb-4B Qwen3.5-4B / JevK5 LoRA 65.4 43 87.9 $0.017 0.2 s Low cost per 1k decisions; single GPU self-host
#15 Clef-Flash Qwen3.5-9B schema head 63.8 42.2 85.5 $0.059 0.4 s Smaller multimodal-trained schema head
Capped Perplexity Decider v1.1 Qwen3.8-27B noncausal 82.8 75.7 89.9 $0.217 0.4 s Unranked (above v1.6.1 open-weights cost cap)
Capped Kev 27B Qwen3.8-27B pointer head 78.9 64.2 93.7 $0.529 0.5 s Unranked (above cost cap); serves /v1/systemone
Capped Clef 27B Qwen3.8-27B schema head 75.3 59.1 91.6 $0.249 0.6 s Unranked (above cost cap); high calibration
02

JevBench v1.4.2.2 composite top ten (September September 28, 2026)

Composite first place: Imajev-4B. Highest intelligence: Jev 1.13.0. Best on the hard tier: Plumb-4B. Fastest: Plumb-4B. No one system leads everything — which is why the composite alone is a poor guide.

RankSystemScoreIntelligenceCalibrationSpeedHard tierSealedRuns as
#1 Imajev-4BMohit Garg · Qwen3.5-4B LoRA 67.4 52.2 80.4 90.6 — 37% Self-hosted GPU
#2 Plumb-4Bcrh225 · JevK5 v0.2 + LoRA 65.8 53 75.5 93.5 77.7% 38% Self-hosted GPU
#3 decider-4b v2Mapika · Qwen3.5-4B 64.1 49.4 75 92.9 67.3% 34.7% Self-hosted, RTX PRO 6000
#4 Jev 1.13.0TypeSafe 63.3 53.1 76.3 83.3 74.1% 36.7% Hosted API
#5 JevK5 v0.2.0allebee · Qwen3.5-4B 62 48.9 74.5 91.1 70% 33.1% Self-hosted, RunPod GPU
#6 Cygnetblockbrain · frozen Gemma 4 12B 61.8 49.5 74.9 90.7 75.5% 33.8% Self-hosted, RTX PRO 6000
#7 HopperHopitAI · Qwen3.5-4B LoRA 59.4 48 79.1 86.8 65% 34.1% Self-hosted, RTX A6000
#8 Winnow-12B Q8EldanRing · Gemma 4 12B 55.6 48.3 64.8 82.3 70.9% 33.1% Self-hosted, 16 GB GPU
#9 reflex 4Bkshetrajna12 54 47.5 70.4 68 63.2% 28.2% Self-hosted GPU
#10 djevMaisa · DiffusionGemma 52.2 47 55.4 91.4 69.5% 29.9% Hosted API, preview

Score, intelligence, calibration and speed are 0–100, higher is better. The hard tier is 220 decisions; sealed is 308 private ones. For Imajev-4B the benchmark publishes the hard tier's two halves — 72.1% public, 75.2% held out — but no combined figure. The fourth axis, cost, is part of the score but not shown; the full table is at the source.

03

Same numbers, different weighting (v1.4.2.2)

The composite weighs speed and cost as heavily as getting the answer right. JevBench also publishes other views of the same measurements — change the weighting and the top of the board reorders.

SystemOfficial compositeAccuracy weighted 60%Intelligence only
Imajev-4B#1#1#3
Plumb-4B#2#2#2
decider-4b v2#3#4#7
Jev 1.13.0#4#3#1
JevK5 v0.2.0#5#6#8

If your pipeline is high-volume and mostly routine, the composite is a fair guide. If the decisions that cost you money are the ambiguous ones, read the accuracy columns instead — that is where Jev's post-training shows up.

04

Reproducible harness & DeepSeek / Qwen / o3-mini comparison

Use this five-point harness before committing a router, then pair Jev with DeepSeek-R1/V3 or o3-mini only for low-confidence escalations. Copy the runnable Python harness script →

CheckSample budgetMeasured metricReproducibility gate
Accuracy (Blind & Sealed) 500+ stratified production events (incl. 20–30% holdout) Exact-match label accuracy & macro-F1 across all classes Pin prompt wording, question definitions, and class order per run
Probability Calibration 10 confidence deciles (0.0–1.0) over binary / choice outputs Expected Calibration Error (ECE) & Brier score Verify 0.85 confidence buckets resolve ~85% true positive rate
Tail Latency (p50 / p95) 200 sequential & concurrent requests from app region End-to-end TLS + API + parse time (ms) Exclude warm-up first call; log timeouts & HTTP 429 separately
Effective Cost / 1k Decisions Measured from input token usage header + downstream escalations Blended $/1k = System-1 input bill + (fallback_rate × System-2 bill) Include output-token charges when comparing chat models
Escalation / Fallback Rate Swept across confidence thresholds τ ∈ [0.75, 0.95] % of decisions routed to GPT / DeepSeek / o3-mini or human queue Report joint accuracy × total blended cost curve per threshold
Model familyArchitectureLatencyCost mechanicsCalibrationProduction fit
TypeSafe Jev (jev-latest / jev-preview) Single-pass System One model; typed choice / score / noul / box JSON (no generated prose) ~0.2 s median (v1.6.1) $0.042 / 1M input tokens ($0.032 / 1k on v1.6.1; $0.0151 / 1k on 350-tok routing), output free 90.6 / 100 on JevBench v1.6.1 out of the box High-volume policy gates, LangChain routers, safety checks, triage & batch scoring
Qwen System-One Fine-tunes (Plumb-4B / Clef / Kev / Decider) Open-weight Qwen3.5 / Qwen3.8 fine-tuned with logit or pointer/schema heads 0.2–0.6 s median self-hosted (v1.6.1) Self-hosted GPU or $0.017–$0.529 / 1k estimated compute on v1.6.1 85.5–93.7 / 100 measured on JevBench v1.6.1 On-prem / VPC workloads where data cannot leave your GPUs
DeepSeek (DeepSeek-V3 / DeepSeek-R1) Autoregressive chat / CoT token generation (<think> + JSON output via chat/completions) Variable by reasoning token length (0.8–8+ s on R1 CoT) Input + generated reasoning/output tokens per request Raw probabilities uncalibrated without post-hoc logprob temperature fitting System-2 escalation for ambiguous edge cases, multi-file synthesis, and freeform replies
OpenAI o3-mini System-2 reasoning model with variable thinking budget & structured JSON mode Multi-second reasoning pass scaling with reasoning_effort Input + reasoning + visible output tokens per call Emits discrete choice; requires external verification / self-consistency for confidence High-stakes fallback path invoked only when Jev confidence stays below threshold
05

How to read the score

  • Composite — harmonic mean of intelligence, calibration, speed and cost, equally weighted. One weak axis pulls it down hard; a cheap fast 4B can outrank a smarter hosted model.
  • Intelligence — how often the system picks the right answer. Sealed decisions make up 20% of this axis.
  • Calibration — whether a stated 0.8 really means about 80%. Critical if you set thresholds like “auto-approve above 0.9”.
  • Tiers — 72 easy, 96 standard, 146 judge-style, 220 hard public cases. Easy and standard separate almost nothing; the hard tier is where systems disagree.
  • Sealed — 308 private cases introduced in v1.4, chance 29.3%. The integrity check on everything above.
06

Head-to-head pages

Each September comparison breaks this board down to one pair. The October pages — Quyet, deck-31B, Perplexity Decider, Kev, Clef, Xor, Jev-Omni, d1-3B, and Plumb — use JevBench v1.6.1 Capability, which is a different headline from the composite on this page.

FAQ

What changed between JevBench v1.4.2.2 and v1.6.1?

September v1.4.2.2 ranked 91 systems using a four-axis harmonic composite (intelligence, calibration, speed, cost). October v1.6.1 focuses on text Capability — combining Intelligence and Calibration subject to an open-weights cost cap — where Jev 1.13.0 is the hosted reference at 77.1 Capability (63.6 Intelligence, 90.6 Calibration, 0.2 s).

What is JevBench?

JevBench is an independent leaderboard for Jev-style System One decision models, published by Benchmark Heaven. Each system answers the same 842 decisions — 534 public across easy, standard, judge and hard tiers, plus 308 sealed cases whose text stays private.

Is Jev the best model on JevBench?

On v1.6.1 Capability, Quyet-1.0-Large (#1, 81.7) and deck-31B (#2, 77.6) lead Jev 1.13.0 (77.1 reference) on raw intelligence, while Jev keeps higher calibration (90.6 vs 90.0 / 82.2) and 0.2 s median latency. On v1.4.2.2, Jev ranks #4 composite (63.3) and #1 intelligence in the top ten.

Why do small open models rank above Jev on v1.4.2.2?

The v1.4.2.2 composite is a harmonic mean of four equally weighted axes: intelligence, calibration, speed and cost. A self-hosted 4B scores very well on speed and API cost, whereas v1.6.1 isolates Capability (Intelligence + Calibration) with an explicit cost gate.

How does Jev compare to DeepSeek, Qwen, and o3-mini?

Qwen-based System-1 models (Plumb-4B, Clef, Kev, Perplexity Decider) compete directly on JevBench v1.6.1. Autoregressive reasoning models like DeepSeek-R1/V3 and OpenAI o3-mini generate variable-length tokens rather than single-pass calibrated logits, making them complementary System-2 escalation targets behind Jev.

Sources