jevmodel.org account

Sign in for 100,000 free input tokens

About 500 Jev requests in the playground or from your own code with an API key. Packs start at $9.90 when you need more. No card required.

By signing in you agree to our terms. We only use your email for your account.
JevBench v1.6.1 methodology · 6 October 2026

Benchmark Jev on the decision, not the demo.

In JevBench v1.6.1, TypeSafe Jev 1.13.0 posts 77.1 Capability (63.6 Intelligence, 90.6 Calibration, 0.2 s median latency) as the hosted reference. Run the reproducible harness below on your own held-out traffic before deciding what routes to System-1 Jev vs System-2 models like DeepSeek or o3-mini.

01

Jev decision accuracy

Compare Jev against the rule, model, or human baseline already in your workflow. A valid label is not automatically the right label.

↗
02

Calibration

Group Jev outputs by confidence and check whether an 0.8 decision is correct roughly 80% of the time on your traffic.

↗
03

Latency

Measure p50 and p95 from your region, including network, gateway, and retries. Vendor 0.2 s figures are a starting point.

↗
04

Fallback rate

Track how often low-confidence Jev outputs still need DeepSeek, o3-mini, GPT, or a human. A cheap model that always escalates is not cheap.

↗
05

Downstream outcome

Measure resolved tickets, accepted routes, prevented risky calls, or another business result. That is the Jev benchmark that matters.

↗
01

JevBench v1.6.1 Capability methodology & reference numbers

Unlike September’s v1.4.2.2 four-axis harmonic composite, JevBench v1.6.1 evaluates eligible models on text Capability — the arithmetic balance of Intelligence (blind + sealed decision accuracy) and Calibration (how faithfully probabilities match real error rates) subject to the open-weights cost cap.

v1.6.1 RankSystemBase / RuntimeCapabilityIntelligenceCalibrationEst. / 1kMedian
Ref Jev 1.13.0 TypeSafe System One (hosted) 77.1 63.6 90.6 $0.032 0.2 s
#1 Quyet-1.0-Large Gemma-4-31B-it (Apache-2.0) 81.7 73.4 90 $0.045 0.4 s
#2 deck-31B Gemma-4-31B-it (self-hosted) 77.6 73 82.2 $0.048 0.4 s
#4 Xor 26B-A4B Gemma-4-26B-A4B MoE 72.4 58.7 86.1 $0.044 0.3 s
#6 Jev-Omni Gemma-4-12B (Apache-2.0) 71.3 55.5 87 $0.029 0.5 s
#12 Plumb-4B Qwen3.5-4B / JevK5 LoRA 65.4 43 87.9 $0.017 0.2 s
#15 Clef-Flash Qwen3.5-9B schema head 63.8 42.2 85.5 $0.059 0.4 s
Capped Perplexity Decider v1.1 Qwen3.8-27B noncausal 82.8 75.7 89.9 $0.217 0.4 s
Capped Kev 27B Qwen3.8-27B pointer head 78.9 64.2 93.7 $0.529 0.5 s
Capped Clef 27B Qwen3.8-27B schema head 75.3 59.1 91.6 $0.249 0.6 s

Source: Benchmark Heaven JevBench v1.6.1 (6 October 2026) ↗.

02

Reproducible production evaluation harness

Run the five checks below on at least 500 blind production cases using fixed question prompts and one threshold sweep.

CheckSample budgetMeasured metricReproducibility gate
Accuracy (Blind & Sealed) 500+ stratified production events (incl. 20–30% holdout) Exact-match label accuracy & macro-F1 across all classes Pin prompt wording, question definitions, and class order per run
Probability Calibration 10 confidence deciles (0.0–1.0) over binary / choice outputs Expected Calibration Error (ECE) & Brier score Verify 0.85 confidence buckets resolve ~85% true positive rate
Tail Latency (p50 / p95) 200 sequential & concurrent requests from app region End-to-end TLS + API + parse time (ms) Exclude warm-up first call; log timeouts & HTTP 429 separately
Effective Cost / 1k Decisions Measured from input token usage header + downstream escalations Blended $/1k = System-1 input bill + (fallback_rate × System-2 bill) Include output-token charges when comparing chat models
Escalation / Fallback Rate Swept across confidence thresholds τ ∈ [0.75, 0.95] % of decisions routed to GPT / DeepSeek / o3-mini or human queue Report joint accuracy × total blended cost curve per threshold
import json, os, statistics, time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["JEVMODEL_API_KEY"],
    base_url="https://jevmodel.org/v1",
)

QUESTIONS = {
    "escalate": {
        "type": "noul",
        "instructions": "Does this ticket require immediate human escalation?",
        "criteria": {
            "true": "Billing dispute, legal threat, data loss, or outage.",
            "false": "Routine how-to question or recoverable self-serve issue.",
        },
    }
}

latencies, briers, hits, escalated = [], [], 0, 0
TAU = 0.80  # Escalate uncertain decisions (< 80% sure) to a System-2 model

with open("eval_cases.jsonl") as f:  # {"state": "...", "label": true}
    rows = [json.loads(line) for line in f if line.strip()]

for row in rows:
    t0 = time.perf_counter()
    res = client.post(
        "/systemone",
        cast_to=dict,
        body={"model": "jev-latest", "state": row["state"], "questions": QUESTIONS},
    )
    latencies.append((time.perf_counter() - t0) * 1000)
    p_yes = float(res["answers"]["escalate"]["noul"])
    y = 1.0 if row["label"] else 0.0
    hits += int((p_yes >= 0.5) == bool(row["label"]))
    briers.append((p_yes - y) ** 2)
    if max(p_yes, 1.0 - p_yes) < TAU:
        escalated += 1

p95 = statistics.quantiles(latencies, n=20)[18] if len(latencies) >= 20 else max(latencies)
print({
    "n": len(rows),
    "accuracy": round(hits / len(rows), 4),
    "brier": round(sum(briers) / len(briers), 4),
    "p50_ms": round(statistics.median(latencies), 1),
    "p95_ms": round(p95, 1),
    "fallback_rate": round(escalated / len(rows), 4),
})
03

Jev vs DeepSeek, Qwen fine-tunes, and o3-mini

General reasoning LLMs (DeepSeek-R1/V3, o3-mini) are evaluated here by architecture, billing mechanics, and System-2 routing role — not synthetic JevBench ranks, because Benchmark Heaven only benchmarks System-1 single-pass decision heads on v1.6.1.

Model familyArchitectureLatencyCost mechanicsCalibrationBest production role
TypeSafe Jev (jev-latest / jev-preview) Single-pass System One model; typed choice / score / noul / box JSON (no generated prose) ~0.2 s median (v1.6.1) $0.042 / 1M input tokens ($0.032 / 1k on v1.6.1; $0.0151 / 1k on 350-tok routing), output free 90.6 / 100 on JevBench v1.6.1 out of the box High-volume policy gates, LangChain routers, safety checks, triage & batch scoring
Qwen System-One Fine-tunes (Plumb-4B / Clef / Kev / Decider) Open-weight Qwen3.5 / Qwen3.8 fine-tuned with logit or pointer/schema heads 0.2–0.6 s median self-hosted (v1.6.1) Self-hosted GPU or $0.017–$0.529 / 1k estimated compute on v1.6.1 85.5–93.7 / 100 measured on JevBench v1.6.1 On-prem / VPC workloads where data cannot leave your GPUs
DeepSeek (DeepSeek-V3 / DeepSeek-R1) Autoregressive chat / CoT token generation (<think> + JSON output via chat/completions) Variable by reasoning token length (0.8–8+ s on R1 CoT) Input + generated reasoning/output tokens per request Raw probabilities uncalibrated without post-hoc logprob temperature fitting System-2 escalation for ambiguous edge cases, multi-file synthesis, and freeform replies
OpenAI o3-mini System-2 reasoning model with variable thinking budget & structured JSON mode Multi-second reasoning pass scaling with reasoning_effort Input + reasoning + visible output tokens per call Emits discrete choice; requires external verification / self-consistency for confidence High-stakes fallback path invoked only when Jev confidence stays below threshold