Benchmark Jev on the decision, not the demo.
In JevBench v1.6.1, TypeSafe Jev 1.13.0 posts 77.1 Capability (63.6 Intelligence, 90.6 Calibration, 0.2 s median latency) as the hosted reference. Run the reproducible harness below on your own held-out traffic before deciding what routes to System-1 Jev vs System-2 models like DeepSeek or o3-mini.
Jev decision accuracy
Compare Jev against the rule, model, or human baseline already in your workflow. A valid label is not automatically the right label.
Calibration
Group Jev outputs by confidence and check whether an 0.8 decision is correct roughly 80% of the time on your traffic.
Latency
Measure p50 and p95 from your region, including network, gateway, and retries. Vendor 0.2 s figures are a starting point.
Fallback rate
Track how often low-confidence Jev outputs still need DeepSeek, o3-mini, GPT, or a human. A cheap model that always escalates is not cheap.
Downstream outcome
Measure resolved tickets, accepted routes, prevented risky calls, or another business result. That is the Jev benchmark that matters.
JevBench v1.6.1 Capability methodology & reference numbers
Unlike September’s v1.4.2.2 four-axis harmonic composite, JevBench v1.6.1 evaluates eligible models on text Capability — the arithmetic balance of Intelligence (blind + sealed decision accuracy) and Calibration (how faithfully probabilities match real error rates) subject to the open-weights cost cap.
| v1.6.1 Rank | System | Base / Runtime | Capability | Intelligence | Calibration | Est. / 1k | Median |
|---|---|---|---|---|---|---|---|
| Ref | Jev 1.13.0 | TypeSafe System One (hosted) | 77.1 | 63.6 | 90.6 | $0.032 | 0.2 s |
| #1 | Quyet-1.0-Large | Gemma-4-31B-it (Apache-2.0) | 81.7 | 73.4 | 90 | $0.045 | 0.4 s |
| #2 | deck-31B | Gemma-4-31B-it (self-hosted) | 77.6 | 73 | 82.2 | $0.048 | 0.4 s |
| #4 | Xor 26B-A4B | Gemma-4-26B-A4B MoE | 72.4 | 58.7 | 86.1 | $0.044 | 0.3 s |
| #6 | Jev-Omni | Gemma-4-12B (Apache-2.0) | 71.3 | 55.5 | 87 | $0.029 | 0.5 s |
| #12 | Plumb-4B | Qwen3.5-4B / JevK5 LoRA | 65.4 | 43 | 87.9 | $0.017 | 0.2 s |
| #15 | Clef-Flash | Qwen3.5-9B schema head | 63.8 | 42.2 | 85.5 | $0.059 | 0.4 s |
| Capped | Perplexity Decider v1.1 | Qwen3.8-27B noncausal | 82.8 | 75.7 | 89.9 | $0.217 | 0.4 s |
| Capped | Kev 27B | Qwen3.8-27B pointer head | 78.9 | 64.2 | 93.7 | $0.529 | 0.5 s |
| Capped | Clef 27B | Qwen3.8-27B schema head | 75.3 | 59.1 | 91.6 | $0.249 | 0.6 s |
Source: Benchmark Heaven JevBench v1.6.1 (6 October 2026) ↗.
Reproducible production evaluation harness
Run the five checks below on at least 500 blind production cases using fixed question prompts and one threshold sweep.
| Check | Sample budget | Measured metric | Reproducibility gate |
|---|---|---|---|
| Accuracy (Blind & Sealed) | 500+ stratified production events (incl. 20–30% holdout) | Exact-match label accuracy & macro-F1 across all classes | Pin prompt wording, question definitions, and class order per run |
| Probability Calibration | 10 confidence deciles (0.0–1.0) over binary / choice outputs | Expected Calibration Error (ECE) & Brier score | Verify 0.85 confidence buckets resolve ~85% true positive rate |
| Tail Latency (p50 / p95) | 200 sequential & concurrent requests from app region | End-to-end TLS + API + parse time (ms) | Exclude warm-up first call; log timeouts & HTTP 429 separately |
| Effective Cost / 1k Decisions | Measured from input token usage header + downstream escalations | Blended $/1k = System-1 input bill + (fallback_rate × System-2 bill) | Include output-token charges when comparing chat models |
| Escalation / Fallback Rate | Swept across confidence thresholds τ ∈ [0.75, 0.95] | % of decisions routed to GPT / DeepSeek / o3-mini or human queue | Report joint accuracy × total blended cost curve per threshold |
import json, os, statistics, time
from openai import OpenAI
client = OpenAI(
api_key=os.environ["JEVMODEL_API_KEY"],
base_url="https://jevmodel.org/v1",
)
QUESTIONS = {
"escalate": {
"type": "noul",
"instructions": "Does this ticket require immediate human escalation?",
"criteria": {
"true": "Billing dispute, legal threat, data loss, or outage.",
"false": "Routine how-to question or recoverable self-serve issue.",
},
}
}
latencies, briers, hits, escalated = [], [], 0, 0
TAU = 0.80 # Escalate uncertain decisions (< 80% sure) to a System-2 model
with open("eval_cases.jsonl") as f: # {"state": "...", "label": true}
rows = [json.loads(line) for line in f if line.strip()]
for row in rows:
t0 = time.perf_counter()
res = client.post(
"/systemone",
cast_to=dict,
body={"model": "jev-latest", "state": row["state"], "questions": QUESTIONS},
)
latencies.append((time.perf_counter() - t0) * 1000)
p_yes = float(res["answers"]["escalate"]["noul"])
y = 1.0 if row["label"] else 0.0
hits += int((p_yes >= 0.5) == bool(row["label"]))
briers.append((p_yes - y) ** 2)
if max(p_yes, 1.0 - p_yes) < TAU:
escalated += 1
p95 = statistics.quantiles(latencies, n=20)[18] if len(latencies) >= 20 else max(latencies)
print({
"n": len(rows),
"accuracy": round(hits / len(rows), 4),
"brier": round(sum(briers) / len(briers), 4),
"p50_ms": round(statistics.median(latencies), 1),
"p95_ms": round(p95, 1),
"fallback_rate": round(escalated / len(rows), 4),
}) Jev vs DeepSeek, Qwen fine-tunes, and o3-mini
General reasoning LLMs (DeepSeek-R1/V3, o3-mini) are evaluated here by architecture, billing mechanics, and System-2 routing role — not synthetic JevBench ranks, because Benchmark Heaven only benchmarks System-1 single-pass decision heads on v1.6.1.
| Model family | Architecture | Latency | Cost mechanics | Calibration | Best production role |
|---|---|---|---|---|---|
| TypeSafe Jev (jev-latest / jev-preview) | Single-pass System One model; typed choice / score / noul / box JSON (no generated prose) | ~0.2 s median (v1.6.1) | $0.042 / 1M input tokens ($0.032 / 1k on v1.6.1; $0.0151 / 1k on 350-tok routing), output free | 90.6 / 100 on JevBench v1.6.1 out of the box | High-volume policy gates, LangChain routers, safety checks, triage & batch scoring |
| Qwen System-One Fine-tunes (Plumb-4B / Clef / Kev / Decider) | Open-weight Qwen3.5 / Qwen3.8 fine-tuned with logit or pointer/schema heads | 0.2–0.6 s median self-hosted (v1.6.1) | Self-hosted GPU or $0.017–$0.529 / 1k estimated compute on v1.6.1 | 85.5–93.7 / 100 measured on JevBench v1.6.1 | On-prem / VPC workloads where data cannot leave your GPUs |
| DeepSeek (DeepSeek-V3 / DeepSeek-R1) | Autoregressive chat / CoT token generation (<think> + JSON output via chat/completions) | Variable by reasoning token length (0.8–8+ s on R1 CoT) | Input + generated reasoning/output tokens per request | Raw probabilities uncalibrated without post-hoc logprob temperature fitting | System-2 escalation for ambiguous edge cases, multi-file synthesis, and freeform replies |
| OpenAI o3-mini | System-2 reasoning model with variable thinking budget & structured JSON mode | Multi-second reasoning pass scaling with reasoning_effort | Input + reasoning + visible output tokens per call | Emits discrete choice; requires external verification / self-consistency for confidence | High-stakes fallback path invoked only when Jev confidence stays below threshold |