On September 15, an AI company called TypeSafe released a model named Jev. Jev doesn't write. It only makes decisions: it picks one of several options, assigns a score, or answers yes or no. On its evals page, TypeSafe also published a leaderboard that ran several models through the same four workflows, such as security incident response, invoice processing, and customer support.
On that leaderboard, Jev made each decision for a fraction of a cent. But trace what TypeSafe treated as the correct answers when it scored those runs, and it becomes clear that the first thing to check when choosing a decision model isn't its price. It's the answer key.
TypeSafe's decision model Jev tied Claude Sonnet 5 at 67.8%, for $0.0004 a decision
Leaderboard scores are the average across the four workflows. Here are Jev and Claude Sonnet 5 side by side.
| Model | Score (average of four workflows) | Cost per decision | Time per decision |
|---|---|---|---|
| Jev | 67.8% | $0.0004 | 0.4 s |
| Claude Sonnet 5 | 67.8% | $0.1174 | 78.1 s |
Nine models appear on the leaderboard. Three score higher than Jev: GPT-5.6 Sol (74.1%), Claude Opus 5 (73.1%), and GPT-5.6 Terra (67.9%). Those three cost $0.0836, $0.1761, and $0.0304 per decision, respectively. Jev comes right after them, tied with Claude Sonnet 5, and it is the cheapest and fastest of the nine. The launch post lists a price of $0.042 per million input tokens, with no charge for output tokens. In effect, the cost of having AI make a decision has dropped to almost zero.
But that 67.8% isn't agreement with answers people set. It's agreement with two AI models that aren't on the leaderboard
What the 67.8% is measured against is spelled out in the scoring method on the same page.
"Instead of debating the correctness of the harness and labels, we assume that the code is correct, and measure against the current smartest large models. For this eval, the reference labels are generated via an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking, answering every question in the harness."
TypeSafe AI, Workflow evals, "How we evaluate"
TypeSafe didn't have people set the correct answers. GPT-6 Astra and Claude Fable 5.1 answered every question, and the average of their answers became the answer key. Neither model is on the leaderboard. So 67.8% is not the rate at which Jev got human-set answers right. It is the rate at which Jev gave the same answer as the two models TypeSafe considers the smartest available today.
TypeSafe itself noted in a footnote to the launch post which way scores built like this lean. Using the average of those two models as the answer key favors OpenAI and Anthropic models, and probably makes Jev and DeepSeek models look worse than they are.
Of 194 projects built with Jev, three put a number on accuracy, and only two checked against answers that didn't come from AI
A site called jevable.com curates projects built with Jev. On September 21, its public API returned 194 projects. Of their descriptions, 27 mentioned a dollar amount and 33 mentioned how long something took. Three gave a number for whether the answers were right.
| What was built | Reported result | What counted as the right answer |
|---|---|---|
| Scam email detection | 96 of 100 correct. Result of the full pipeline, which passed ambiguous emails to another model | 100 emails with known answers |
| Postgres query planner | Queries ran 12% faster | Actual query times in a benchmark |
| Construction drawing classification | 100% agreement | Answers from GPT-4.1 and GPT-6 Astra |
The scam email detector and the query planner got their answers from somewhere other than AI output. The construction drawing classifier did what the leaderboard did: it treated another AI model's answers as correct.
A tool on the same list, jevcal, opens its description with "Everyone picks Jev confidence thresholds by vibes. 0.95? 0.5?" Jev returns a confidence score with each decision, and the point is that people decide by gut feel how high that score has to be before they trust the decision. jevcal says that if you give it your own data and the accuracy you want, it will set that threshold for you. OpenChamber, which collected and analyzed tweets from Jev's first four days, took up the same question. It wrote that the score alone can't tell you whether 0.95 means 95% correct on your own work. You only find out by checking against examples with known answers.
When Red Hat retested on public data that came with answers attached, Jev ranked first on one task and fourth on the other
On October 2, Rob Geada, Mac Misiura, and Shelton Cyril of Red Hat published a post on Red Hat Developer comparing decision models like Jev with existing AI safety filters, or guardrails (updated October 5). There were two tasks: blocking prompt injection, meaning instructions written to trick an AI, and filtering harmful content. The three built guardrails in nine configurations across four families, then had each one judge public datasets in which risky and safe text already carried correct labels. The data was split evenly between risky and safe, and each method's block-or-allow decision was compared with the dataset's label.
| Task | Rank | Method | Accuracy |
|---|---|---|---|
| Blocking prompt injection | 1st | Qwen3.6-35B (a general model used as the judge) | 89.31% |
| Blocking prompt injection | 2nd | Red Hat pretrained classifier (200 million parameters) | 89.01% |
| Blocking prompt injection | 4th | Jev | 86.35% |
| Filtering harmful content | 1st | Jev | 86.20% |
| Filtering harmful content | 2nd | DiffusionGemma | 85.53% |
| Filtering harmful content | 3rd | Qwen3.6-35B | 85.47% |
Jev didn't lead on speed either. On prompt injection, the Red Hat classifier's latency was 54.1 ms, while Jev's was 348.1 ms. The three wrote that they found no evidence that decision models were faster, cheaper, or better than having a general model act as the judge.
"we did not find that decision models produced faster, cheaper, or higher-quality answers compared with LLM-as-a-judge."
Rob Geada, Mac Misiura, Shelton Cyril, Red Hat Developer
This comparison has limits too. The authors ran the tests in the UK and noted themselves that round trips to the Jev API and Red Hat's cluster in the US added at least about 56 ms. There were only two tasks, and only English text was used. And the pretrained classifier that came second is the model Red Hat ships as a default guardrail in its own product, OpenShift AI 3.6.
On TypeSafe's leaderboard, Jev came right after three of the nine models, tied with Claude Sonnet 5, and the answer key there was the output of two AI models. In Red Hat's comparison, the answer key was the labels the datasets came with, and Jev ranked first on filtering harmful content and fourth on blocking prompt injection. The two leaderboards also cover different tasks, so the change in ranking can't be pinned on the answer key alone. Still, in a comparison where someone else set the answers, the launch post's claim of similar performance at far lower cost and far higher speed did not hold up.
When a decision costs $0.0004, the first question in choosing a decision model isn't price. It's whether your work has an answer key
In the launch post, TypeSafe describes GPT-6 Astra and Claude Fable 5.1, the two models behind the leaderboard's answer key, as the largest, smartest, and most expensive models made by other companies. Every ranked model answered with its provider's default settings. Only the two models that produced the answer key answered with long thinking turned on.
Even the company selling a model that makes a decision in 0.4 seconds for $0.0004 built the answer key for grading those decisions by having the two most expensive models think at length. When Red Hat, which had data with answers already attached, compared the same model again, the rankings changed from task to task.
Now that there's a model that can make a decision for $0.0004, the first thing to check when choosing a decision model isn't the price. It's whether you have an answer key that can tell you if those decisions were right for your work.
Going deeper
The scoring method and the four workflows How the answer key was built and which workflows separated the models, with per-workflow tables evals.typesafe.ai
A guardrail comparison on labeled data Red Hat's own write-up of results and latency for all nine methods, plus the limits of the test setup developers.redhat.com
A tally of 12,759 tweets What you need to check whether a confidence score matches how often the answers are actually right openchamber.dev