# Benchmarking decision models before trusting them

**Topic:** LLM evals · **Published:** October 2026 · **Reading time:** 8 min

In September 2026 I tested a small decision model as the judgment layer of an AI agent fleet. It decided which agent takes each request, how risky the request is, and whether to hold it for a check before any agent writes code.

The headline result was strong: 16 of 16 correct, against 13 of 16 for a frontier model. The more useful results were the ones that went wrong first. This article covers the benchmarks, the mistakes in my own measurements, and the rules I now apply to any evaluation.

## The model

Jev (TypeSafe) is a typed judgment model. It does not write text. It answers one question at a time from a fixed set of options (multiple choice or yes or no) and returns a confidence for its answer. I used it in three places:

- **Classification:** what kind of request this is, and how risky.
- **Risk checks:** yes-or-no questions such as "does this change send data to an outside service?"
- **Routing:** which agent owns the request, and whether to hold it.

## Result 1: classification

On 16 hand-labeled prompts, Jev labeled all 16 correctly. A frontier model labeled 13 correctly (0.813).

| | Jev | Frontier model |
| --- | --- | --- |
| Correct | 16 of 16 | 13 of 16 |
| Mean wait | 919 ms | 8,966 ms |
| Tokens per case | 794 | 21,167 |

Read the token figure with care. Most of the 26× gap came from the 20,000-token system prompt the frontier model carried in that setup, not from the task itself. Sixteen cases also make a small sample. The result showed that the small model was good enough for this job, not that it is better in general.

## Result 2: risk checks, and a holdout that changed the story

I tuned the wording of each risk question on a set of known cases. The tuned questions agreed with my labels on 93% of them.

That number was in-sample: I had chosen the wording on the same cases I reported. So I froze the wording and wrote a new holdout set after the freeze. Agreement dropped to 77–86%. Worse, it had one false negative that repeated 30 times out of 30: a change that sent a customer's email and postal address to an analytics vendor passed as safe.

For a safety check, the gate is zero false negatives, not a high average. I added explicit wording for the missed case ("sends any data to an external service, vendor or off-site destination") and wrote a new, fresh holdout of 10 cases, because a holdout you have already used is no longer a holdout. The new wording caught every hazard in it. Each check cost about 1/600 of a mid-tier model call, with a mean wait of 345 ms against 15.7 s.

## Result 3: routing, quoted as a range

The router picked the owner of each request. I ran it four times on the same frozen set of cases, with no change in between. It scored 74.3% to 76.3% (one earlier run reached 79.6%). The model is not deterministic, so I quote about 76% ± 3, never a single run.

Inside that number:

- **Owner choice** was correct on 33 of 33 scored cases.
- **Unsafe permissions** were zero in every run. In every run, the router granted no forbidden write.
- **Holds** were the weak part. Only 28.6–31.6% of holds were needed, so about two in three held requests waited for nothing.

The zero unsafe permissions and the unneeded holds came from the same design: when unsure, hold. That is the right default for safety, and it has a cost in speed. About two weeks later the router was retired, and the measurement method stayed.

## Mistakes in my own measurements

**A baseline I forgot.** One rule set had "zero under-classified cases across 403 real cases". That was true and meant nothing. The rule could only answer the top risk tier, so it could never under-classify. A constant answer of "top tier" matched 59% of all cases. The rule matched 53% of the cases it covered. Without the baseline, 53% looked like progress.

**Evidence that raised confidence and lowered safety.** I added a keyword scan as extra evidence for the question "does this change have side effects that cannot be undone?" The model went from answering 2 of 403 cases to 265 of 403, and every one of the 265 answers was "no". A keyword scan cannot tell a change that mentions email from a change that sends email. More answers looked like progress, but all the new answers were in the unsafe direction. I reverted it. Measure the direction of a change against the safe default, not only how many cases get an answer.

**A spelling defect that made a phantom result.** One label was spelled `verify_after` in the data and `verify-after` in the options. 164 of 384 labels could not match, and one question looked like 3% agreement. After the fix it was 71%. The lesson: fail the build when a label is not in its option set.

**Small samples.** On an open-weights model, four runs of a puzzle suggested maximum reasoning effort beat low effort (3 of 4 against 2 of 4). Ten runs gave 6 of 10 at both. Four runs on a noisy task cannot tell 50% from 75%.

## Rules I now apply to any evaluation

1. **Report the trivial baseline beside every headline number.** If a constant answer passes your test, the test is not evidence.
2. **Freeze the setup, then write the holdout.** Tuning on the cases you report is in-sample.
3. **Use a fresh holdout after every fix.** A used holdout has lost its value.
4. **Set the gate by the cost of an error.** For a safety check, one false negative fails it, whatever the average.
5. **Quote non-deterministic results as a range,** over repeated runs on the same data.
6. **Check the direction, not only the rate.** More answers can mean more wrong answers.
7. **Validate labels in the build.** A typo can create or hide a result.

Want evals like these for your own agents or LLM features? Email [hello@jeffkent.dev](mailto:hello@jeffkent.dev).