Benchmarking decision models before trusting them

In September 2026 I tested a small decision model as the judgment layer of an AI agent fleet. It decided which agent takes each request, how risky the request is, and whether to hold it for a check before any agent writes code.

The headline result was strong: 16 of 16 correct, against 13 of 16 for a frontier model. The more useful results were the ones that went wrong first. This article covers the benchmarks, the mistakes in my own measurements, and the rules I now apply to any evaluation.

The model

Jev (TypeSafe) is a typed judgment model. It does not write text. It answers one question at a time from a fixed set of options (multiple choice or yes or no) and returns a confidence for its answer. I used it in three places:

• Classification: what kind of request this is, and how risky.
• Risk checks: yes-or-no questions such as “does this change send data to an outside service?”
• Routing: which agent owns the request, and whether to hold it.

Result 1: classification

On 16 hand-labeled prompts, Jev labeled all 16 correctly. A frontier model labeled 13 correctly (0.813).

Read the token figure with care. Most of the 26× gap came from the 20,000-token system prompt the frontier model carried in that setup, not from the task itself. Sixteen cases also make a small sample. The result showed that the small model was good enough for this job, not that it is better in general.

Result 2: risk checks, and a holdout that changed the story

I tuned the wording of each risk question on a set of known cases. The tuned questions agreed with my labels on 93% of them.

That number was in-sample: I had chosen the wording on the same cases I reported. So I froze the wording and wrote a new holdout set after the freeze. Agreement dropped to 77–86%. Worse, it had one false negative that repeated 30 times out of 30: a change that sent a customer’s email and postal address to an analytics vendor passed as safe.

For a safety check, the gate is zero false negatives, not a high average. I added explicit wording for the missed case (“sends any data to an external service, vendor or off-site destination”) and wrote a new, fresh holdout of 10 cases, because a holdout you have already used is no longer a holdout. The new wording caught every hazard in it. Each check cost about 1/600 of a mid-tier model call, with a mean wait of 345 ms against 15.7 s.

Result 3: routing, quoted as a range

The router picked the owner of each request. I ran it four times on the same frozen set of cases, with no change in between. It scored 74.3% to 76.3% (one earlier run reached 79.6%). The model is not deterministic, so I quote about 76% ± 3, never a single run.

Inside that number:

• Owner choice was correct on 33 of 33 scored cases.
• Unsafe permissions were zero in every run. In every run, the router granted no forbidden write.
• Holds were the weak part. Only 28.6–31.6% of holds were needed, so about two in three held requests waited for nothing.

The zero unsafe permissions and the unneeded holds came from the same design: when unsure, hold. That is the right default for safety, and it has a cost in speed. About two weeks later the router was retired, and the measurement method stayed.

Mistakes in my own measurements

A baseline I forgot. One rule set had “zero under-classified cases across 403 real cases”. That was true and meant nothing. The rule could only answer the top risk tier, so it could never under-classify. A constant answer of “top tier” matched 59% of all cases. The rule matched 53% of the cases it covered. Without the baseline, 53% looked like progress.

Evidence that raised confidence and lowered safety. I added a keyword scan as extra evidence for the question “does this change have side effects that cannot be undone?” The model went from answering 2 of 403 cases to 265 of 403, and every one of the 265 answers was “no”. A keyword scan cannot tell a change that mentions email from a change that sends email. More answers looked like progress, but all the new answers were in the unsafe direction. I reverted it. Measure the direction of a change against the safe default, not only how many cases get an answer.

A spelling defect that made a phantom result. One label was spelled verify_after in the data and verify-after in the options. 164 of 384 labels could not match, and one question looked like 3% agreement. After the fix it was 71%. The lesson: fail the build when a label is not in its option set.

Small samples. On an open-weights model, four runs of a puzzle suggested maximum reasoning effort beat low effort (3 of 4 against 2 of 4). Ten runs gave 6 of 10 at both. Four runs on a noisy task cannot tell 50% from 75%.

Rules I now apply to any evaluation

1. Report the trivial baseline beside every headline number. If a constant answer passes your test, the test is not evidence.
2. Freeze the setup, then write the holdout. Tuning on the cases you report is in-sample.
3. Use a fresh holdout after every fix. A used holdout has lost its value.
4. Set the gate by the cost of an error. For a safety check, one false negative fails it, whatever the average.
5. Quote non-deterministic results as a range, over repeated runs on the same data.
6. Check the direction, not only the rate. More answers can mean more wrong answers.
7. Validate labels in the build. A typo can create or hide a result.

Want evals like these for your own agents or LLM features? Email [email protected].