# Jeff Kent Buenaflor: case studies and articles > Principal AI engineer who designs and runs AI agent teams that build, test and review production software. Remote, from the Philippines (UTC+8). Contact: hello@jeffkent.dev ## Highlights - About 31x value versus spend, Feb to Oct 2026: about $124,000 of AI work at API list prices for about $4,000 in flat-rate Claude and Codex seats (personal and employer-paid), 235 billion tokens. - Decision models tested before trusted: a typed judgment model for routing and risk checks scored 16 of 16 versus 13 of 16 for a frontier model, at 26x fewer tokens. Retired after measuring that about two in three of its holds were unneeded. - About 20x lower cost per coding task: benchmarked models and settings, then routed each task to the cheapest option that met the review bar, with no rise in review failures. Open-weights workers (DeepSeek V4.1 Flash, GLM 5.3 Flash, via Ollama and direct APIs) ran under frontier reviewers. - Process: a planner splits each task, low-cost builders implement, an independent reviewer must approve every change, then the result is verified in the real environment. --- Source: https://jeffkent.dev/case-studies/fintech/ # Case study: rebuilding a regulated fintech platform with a team of AI agents **Role:** CTO and hands-on architect · **Context:** regulated crypto buy-and-sell company, Philippines · **Period:** 2026 · **Status:** in final testing ## The problem The company needed its whole platform rebuilt: 9 apps, including customer sign-up, identity checks, crypto buy and sell, deposits and withdrawals, and the staff back office. The work touched money, identity verification and sign-in, where a careless change becomes a regulatory problem, not just a bug. My goal was to make AI agents do most of the building, while keeping the review discipline a regulated company needs. ## My role I set the architecture and the acceptance criteria, and agents wrote most of the code under my review gate. My own architecture decisions: a NestJS modular monolith with 14 domain modules on PostgreSQL, Redis and Cognito; typed API contracts (ts-rest and zod) shared by the back end and 5 React web apps; money held in integer centavos until display; role separation and break-glass access on the money path, with two-person approval for compliance, fee and rate changes; a separate rate-polling worker per price source; and Terraform on AWS ECS, reachable only through Cloudflare. Every production deploy and every business-logic change needs my approval. Deterministic checks (tests, type checks, builds, scans) run automatically. ## What I built ### One job per agent Each agent gets a single role: plan, build, review or test. A planner splits each ticket into small, clear items. A builder takes one item with a short, written brief: the files it may touch, the goal, and the checks that prove it is done. A builder never reviews its own work. At peak, more than 45 agents worked at the same time across Claude, Codex and Grok, as long-running lanes plus short-lived helpers. ### Model routing by risk and cost I benchmarked models and settings on real tasks for cost and quality, then routed each job to the cheapest option that met the quality bar: - **Small model, default effort:** mechanical, tightly scoped work such as one-file fixes, fixed checklists and scripted releases. Raising its effort to maximum cost far more with no gain in quality. - **Mid-tier model:** normal feature builds and repairs, and items the small model failed twice. - **Strongest model, high effort:** planning, every independent review, and anything that touches money, identity checks, sign-in, security, migrations or many files at once. The result was about a 20-fold lower cost per coding task, with no rise in review failures. ### Measuring decision models before trusting them I tested Jev (TypeSafe), a typed judgment model that answers multiple-choice and yes-or-no questions, as a decision layer: which agent takes a request, how risky it is, and whether to hold it for a check. - **Classification:** it labeled 16 of 16 hand-labeled prompts correctly, against 13 of 16 for a frontier model, with 26× fewer tokens (mostly because the comparison model carried a 20,000-token system prompt) and about a tenth of the wait. - **Risk checks:** wording tuned on known cases scored 93% agreement, but a frozen holdout dropped to 77–86% with one repeated miss. A rewritten version then passed a fresh 10-case holdout with no misses, at about 1/600 the cost of a mid-tier model per check. - **Routing:** 76% ± 3 accuracy over four runs, with correct owner assignment on 33 of 33 scored cases. I retired it after about two weeks, after measuring that about two in three of its holds were unneeded. What stayed is the method: freeze a holdout, report the trivial baseline, and repeat runs before you believe a gain. ### Open-weights models as workers I ran open-weights models next to the commercial ones. DeepSeek V4.1 Flash did bounded build, rewrite and review work through Ollama and the DeepSeek API, including as the model behind Claude Code, with about six workers in parallel under the host's concurrency cap. GLM 5.3 Flash handled scoping and build steps in a separate workflow. In a ten-run test, DeepSeek's maximum reasoning effort solved a test puzzle no more often than its lowest setting (6 of 10 at both), although a four-run sample had suggested otherwise. ### Subscriptions, not metered API I ran the Claude and Codex agents on flat-rate subscriptions, not pay-per-token API keys. From February to October 2026, across my Claude and Codex agent work (this platform included), the agents used 235 billion tokens (197 billion in 1.5 million Claude messages, 38 billion on Codex). At API list prices that is about **$124,000**, against about **$4,000** in subscriptions (one personal Claude plan until September, one Codex plan from August, then nine employer-paid Claude seats in October), so about 31 times the value at list prices. In October alone, $48,500 of usage ran on $2,000 of seats. About 97% of the Claude tokens were cache reads of earlier context, which is why trimming each agent's context mattered. ### Independent review on every change Every change needs an approval from a separate reviewer agent, made against the exact version being merged, on top of the automated gates. Every release to customers also needs a human approval. Reviews of user-facing work include screenshots and a checklist of small details (wording, spacing, empty, loading and error states), because many small misses add up to a poor product. Approved changes land in batches, and a staging build is verified after each batch. ### Code quality the build enforces, not reviewers AI agents copy code readily, so a fast agent team can produce a large, repetitive code base. For the rewrite of the staff back office front end, I made reuse a rule that the build checks: - One shared component kit (58 parts so far: tables, drawers, dialogs, status chips, empty and error states). A missing pattern goes into the kit first, with its own tests, before any page uses it. - Pages cannot add their own styles. Colors, spacing and type sizes are tokens defined once, and status and error wording is defined once per value. - Any copied block of 50+ tokens, any unused file or export, and any file over 200 lines fails the build. The app it replaces, area by area, had a single file of 7,000 lines. A reviewer still requests changes for copied markup or a one-off version of a kit part, but the build catches most of it first. ### Safety boundaries Agents cannot delete files, send email, or change the live system. Every release to customers needs a human approval, and the staging and production environments are always named in every report, so a staging result can never be mistaken for a production one. A test that runs against mocks is never accepted as production evidence. ### Shared memory that survives sessions and providers Agents forget everything between sessions, and different providers' tools do not share state. I built a shared memory of 600+ short notes: rules, decisions and lessons from past incidents, each with the reason behind it. Agents read the relevant note before they act, and add one after a verified outcome. ### Delivery infrastructure A scheduler spreads builds and tests across three machines (196 jobs in one day) and caps heavy jobs so agents do not starve each other. The test suite includes 1,000+ Playwright end-to-end tests, and a Telegram bot connects 33 team chats to the agents so staff can raise requests where they already work. ## Results - About **4,000 approved code changes in under six months** across 9 apps (TypeScript, NestJS, React, PostgreSQL, Terraform on AWS). - **3,000+ automated test files**, and an independent approval on every change. - A back office front end rebuilt on **58 shared components**, where copied code, unused code and oversized files fail the build. - About **20-fold lower cost per coding task** after routing work by risk and cost. - About **$124,000 of AI usage at API list prices on about $4,000 of flat-rate subscriptions** (personal and employer-paid), February to October 2026, across all my agent work. - The platform replaces the company's current platform and is in final testing. The same approach let me rebuild a streaming service as its only engineer: a viewer website, a staff back office with its back end, and native iOS and Android apps with 600+ tests, now in testing. ## Lessons learned - **Fewer, stronger agents beat more agents.** At peak, 45+ agents ran at once, with separate lanes for features, bugs, backlog, review and chat. As models improved, a strong planner needed far shorter instructions, so I closed most of those lanes. Today 10+ long-running lanes do the work: one planner splits each ticket into small items for low-cost builders, and independent reviewers check every merge. - **Report the trivial baseline.** A risk-tier rule matched the recorded tier on 53% of the cases it covered, while always answering the top tier matched 59% of all cases. Without the baseline, 53% looked like progress. - **Context is the main cost.** Agents that reload long histories spend most of their budget re-reading. Each agent now starts with only the brief it needs. - **Agents must fail loudly.** An agent waiting on a prompt nobody sees blocks its queue in silence. Agents now stop and report a blocker instead. - **Set scope and effort per task.** Unscoped agents wander, and maximum effort on small tasks costs more without better results. Every brief names its files, its goal and its stop point. - **Every helper needs an end condition.** Tunnels, dev servers and watchers outlive the agents that start them unless each job ends with a cleanup check. ## What I bring to your team - A working model for AI-assisted delivery that keeps review, security and release controls intact. - Measured cost and quality decisions, not guesses about which model to use. - Hands-on engineering across TypeScript back ends, React, native mobile, AWS and Terraform, with deep experience in regulated money and identity flows. - Each client's code stays in that client's own repositories. --- Source: https://jeffkent.dev/case-studies/streaming/ # Case study: orchestrating AI agents to rebuild a streaming platform on web, iOS and Android **Role:** lead engineer (sole engineer), contract · **Context:** a Southeast Asian video streaming service (movies, series, live TV and sports) · **Period:** Aug 2026 – Present · **Status:** in testing ## The problem The service needed its whole platform rebuilt: a viewer website, a staff back office for the catalog, live channels and advertising, the APIs behind both, and native iOS and Android apps. That is normally the work of a team with web, back-end and two mobile specialists. I was the sole engineer, so the real job was to design an agent team that could do the work and a system that could prove the work was right. ## The agent setup ### I plan, agents build I set the architecture, split the work into small items and decide what counts as done. Agents from two providers (Claude and Codex) build the items. Each agent works in its own branch and worktree, so agents never edit the same files at the same time. Agents wrote nearly two-thirds of the web monorepo's commits. ### One quality gate, the same for everyone Every change to the web monorepo must pass one check before it lands: a frozen dependency install, lint, type check, tests, builds, unused-code detection, import-boundary rules and a secret scan. My changes and the agents' changes go through the same gate. ### A second agent checks before tests run The native apps are tested in rounds against the real staging back end. Before a round runs, a second agent checks its plan. Each round records the commit hash of the exact code under test, takes a fresh snapshot of the test account's data, and cleans up every simulator and process it used. A result that cannot be traced to the exact code does not count. ### Agents share devices by handoff iOS and Android test runs share one test machine. An agent must be handed a device and must hand it back before another agent can use it, so two agents never fight over one simulator. ## What the agents delivered - **Web platform:** a viewer website, a viewer API, a staff back office and its API in one TypeScript monorepo (React, Fastify, PostgreSQL, AWS). The back office covers live operations, catalog curation and publishing, artwork uploads, captions and subtitles, advertising (VAST ad tags), customers, access groups and roles, account security, the TV guide, analytics and an audit trail. - **iOS (Swift, SwiftUI):** the website's main viewer features, including on-demand and live playback, native ads, caption and subtitle choice, My List and Continue Watching, and calendar reminders. 272 unit tests and 76 UI tests. - **Android (Kotlin, Jetpack Compose):** playback with Widevine DRM, WebVTT subtitles, a 7-day TV guide, IMA ads with consent, picture-in-picture, and session data encrypted with an Android Keystore key. About 330 test functions. ## Results - One engineer and an agent team have rebuilt a streaming platform across **web, back end, iOS and Android** in about ten weeks. It is now in testing. - **600+ automated tests** written for the native apps, plus the monorepo's test suites, and one quality gate on every monorepo change. - The staging test rounds found real defects before users could, and each one went into the fix queue, not to users. - An unsigned iOS release archive builds, and an Android release build has been run. Store release, physical-device testing and full ad-completion testing are still ahead. ## Lessons learned - **Parallel agents need fully separate state.** Separate branches are not enough if agents share any tool state, so builders also avoid shared scratch areas like git's stash. - **Heavy test tooling belongs on its own machine.** Simulators compete with agents for disk and CPU, so all simulator work runs on a dedicated test Mac and shuts down after each check. - **A test result must prove what it tested.** Commit hashes, fresh test-account snapshots and explicit device handoffs were part of the design from the start. ## What I bring to your team I design agent teams and the checks around them, so a very small team can build a product across web, back end, iOS and Android with the checks a larger team would use, without lowering the bar for what counts as done. The client's code stays in the client's own repositories. --- Source: https://jeffkent.dev/case-studies/media/ # Case study: an AI agent studio that writes, renders and schedules short-form video **Role:** sole builder, personal venture · **Context:** faceless short-form channels (food history and daily AI news) on Instagram, YouTube, TikTok and Facebook · **Period:** Aug 2026 – Present · **Status:** in production since Oct 2026 ## The problem A short educational video needs research, a script, a voice, music, images, motion, captions, an edit, a loudness check and an upload to each platform. A small studio does this with a writer, an editor, a motion designer and a social media manager. I wanted to find out how much of that chain AI agents could run end to end, at a quality bar I would accept, while I only made the creative calls. ## The agent setup ### Three agents, three jobs - **Director (Claude Opus 5.5).** Owns the series bible, script, look, shot list, visual review and final edit. It writes each step as a numbered runbook. - **Executor (Codex, gpt-6.1-sol).** Runs the runbooks: image generation, render jobs, audio mix and quality checks. It stops at the first mismatch and reports, not guesses. - **Operator (Codex, gpt-6-luna with computer use).** Drives the browser to upload, schedule and set thumbnails on each platform, then records what the platform confirmed. ### One manifest per episode Each episode is one manifest file. A new series needs only a new series bible and new manifests; the pipeline does not change. ### The pipeline | Stage | Tools | |---|---| | Facts and script | Director agent; an independent agent fact-checks before anything is published | | Voice, music, sound effects | ElevenLabs (eleven_v4 voice, a new instrumental track per episode) | | Stills | Codex image generation, in a real phone-photo look | | Motion | Higgsfield (Kling 3.0) clips only where motion adds value; depth maps and headless Blender 5 for 2.5D camera moves | | Edit and captions | Remotion 4 design system, word-timed captions | | Mix, master, quality check | ffmpeg; loudness, true peak, duplicate-music and layout checks | | Publish | Browser agent schedules each platform from one post queue with file checksums | Renders run on a separate Windows GPU machine, so the laptop stays free. ## Results - **28 finished episodes in about two weeks** (Oct 12–25, 2026), each one passing an automated 12- to 13-point quality check before scheduling. - **53 uploads scheduled** on YouTube and Instagram from one queue, with TikTok and Facebook cuts from the same masters. - **Paid video kept small:** a 5-second Kling clip costs 8.75 credits, so the pipeline uses stills and 2.5D motion by default and a paid clip only where it earns its place. - **A second series in days:** a daily AI-news format (three Shorts a day with an AI avatar host and an on-screen "AI host" label) reused the same voice, render and upload stages with a new Remotion template. ## Lessons learned - **Make the brand visible on a phone, not on a monitor.** The first logo was 5.8% of the frame width and disappeared on a phone. It became a sticker at 12–14% of the width, placed clear of the Reels and TikTok buttons. - **Check the music, not only the voice.** Early beds sat 15 dB under the voice and one track was reused. Each episode now gets its own track, a checksum blocks repeats, and the bed sits about 10 LU under the voice. - **Render one at a time when disk is the limit.** Three masters rendered at once filled the render disk, and two failed. The render chain now starts a job only when at least 15 GB is free. - **Browser agents need short, unambiguous paths.** The upload dialog joined a new path to the last one, so one episode's file went up under another episode's name. Files now go into short, single-purpose folders and every upload is checked against its checksum. - **One blocking review is enough for video.** Several review rounds slowed delivery without better results. Each episode now gets one blocking check plus the automated quality gate. - **Generated faces are the weak point.** All five clips for one trailer were rejected because the video model redrew the faces. Clips are now used for food, objects and places, not for characters' faces. ## What I bring to your team I can design and run a multi-agent media pipeline end to end: script, voice, image, video, edit, quality checks and publishing, with the agents' roles, checks and costs set out in writing. The same pattern (director, executor, operator, one manifest, one quality gate) carries over to marketing content, product videos and documentation. --- Source: https://jeffkent.dev/writing/45-agents-to-10/ # How I cut 45 AI agents to 10+ long-running lanes **Topic:** agent orchestration · **Published:** October 2026 · **Reading time:** 8 min At its peak in 2026, more than 45 AI agents worked at the same time on one product: a regulated crypto platform with 9 apps that I was rebuilding as CTO. Some were long-running agents with fixed jobs. The rest were short-lived helpers. They ran on Claude, Codex and Grok. By late October, 10+ long-running agents did the same work. Over the whole project, the team shipped about 4,000 approved code changes in under six months, each one approved by an independent reviewer agent. This article explains why the large team stopped working, what replaced it, and the numbers behind each step. ## Why I scaled up first Earlier models needed long, exact instructions. A builder agent that knew only a ticket number made poor guesses about files, patterns and rules. So I gave each agent a narrow area and kept it there for weeks: one for features, one for bugs, one for small fixes, one for finance screens, one for accounting, several for review, several for QA, one to talk with the staff in their chat groups, and one for tooling. Specialists learned their area. Adding an agent felt like adding capacity, the same way adding an engineer does. ## What broke A team of 45 agents has the same problems as a team of 45 people, and agents do not notice the problems themselves. By the end of September, these problems had shown up: - **Steps with two owners, or none.** The lead, the QA agent and the builders all moved tickets after a deploy. One agent both scoped requests and assigned them, so nobody checked its scope. - **Builders without a reviewer.** Two builder agents had no reviewer assigned, so their changes waited, then bounced. - **Stale rules.** One builder still followed a check that I had already retired, because its instructions were older than the change. - **Collisions.** Agents in different areas edited the same shared files and blocked each other's merges. - **Machine load.** Type checks and test suites from many agents at once starved the build machines. None of these was a model failure. Each model did its job well. The failures were in coordination, and coordination cost grows faster than the team. ## Step 1: one owner per step I wrote a table with one row per step of the delivery flow and exactly one owner per row: intake, scope, assign, build, review, land, verify, retest, team relay and tooling. An agent that gets a request outside its row sends it to the owner and does not do it. The table gave each problem on the list one owner. It also showed that many lanes existed only because nobody had asked whether their step needed a dedicated agent. ## Step 2: fewer builders, bigger batches Next I limited new work to four builder lanes, split by risk area: compliance and sign-in, tooling and tests, product screens, and finance. The other builders finished their open changes, then stopped. Builders also moved from one ticket per change to batches of 5 to 50 related tickets per change. One batch needs one review and one merge, not fifty. ## Step 3: builders became small and cheap The biggest cut came from a model change. In October I ran four pilots with a small, cheap model (Claude Haiku 5.5) as the builder. A strong planner (Claude Opus 5.5) split each ticket into small items with exact briefs: the files, the shared components and their limits, and the checks that prove the item is done. An independent Opus reviewer approved or rejected the merged result. | Pilot | What changed | Round-1 misses | Rounds to approval | Builder cost | | --- | --- | --- | --- | --- | | 1 | Haiku replays 3 shipped tickets | 7 | 1–2 | $0.02–0.43 per ticket | | 2 | Planner splits 2 tickets into 45 items | 4 | 2 and 3 | $0.65 in total | | 3 | Same as 2, at maximum reasoning effort | 5 | 2 and 2 | about 20× pilot 2 | | 4 | Same harness, 41 items, three builders | 4 / 5 / 6 | 2–3 | $0.015 / $0.306 / $0.238 per item | Pilot 4 compared Haiku at default effort, Haiku at maximum effort and Sonnet 5.5 at high effort on the same items. The cheapest option had the fewest misses. Maximum effort cost about 20 times more, took about 9 times longer and missed more. The reviewer caught the misses in every arm. The pilots showed where the misses came from: mostly the planner and the briefs, not the builder's model or effort. The planner forgot pages that used the same pattern, and briefs left out limits of shared components. So I added two rules: every plan has a "same pattern elsewhere" item, and every brief names the shared-component limits. With that result, a builder no longer needs a strong model or a long memory. In two days, 15 standing builder lanes closed. Four small-model builder lanes replaced them, and each one takes short, briefed items from the lead. A newer planner lane starts a short-lived builder for each item. ## Step 4: remove extra layers I also removed two layers that I had built myself: - **A decision-model router.** A typed judgment model decided which agent took each request and whether to hold it for a check. It was accurate on owner choice, but about two in three of its holds were unneeded. I describe that benchmark in a [separate article](/writing/benchmarking-decision-models/). - **A second-model-family review lane.** For one day, Codex agents did reviews, so a different model family checked Claude's work. The next day I moved all review back to the Opus reviewers. ## What the 10+ lanes look like now - **Planners and small-model builders.** The lead or a planner splits each ticket into small, briefed items. Four small-model builder lanes take them one at a time, and one planner starts a short-lived builder for each item. - **Independent reviewers.** A reviewer never wrote the code it reviews. Changes that touch money, identity checks, sign-in, security or migrations stay with the strongest model, as builder and as reviewer. - **One lead.** It orders the queue, settles collisions and lands approved changes in batches. A staging build is verified after each batch. - **A few service lanes.** Team intake, QA and tooling. Hard limits keep the machines healthy: at most three heavy jobs at once on the main build server, one build worker per lane, and at most four background helpers per lane. When an account reaches 90% of its five-hour usage window, a guard holds new helpers until the window resets. ## What I would tell a team starting with agents 1. **Count coordination cost, not agent count.** Every extra standing agent adds handoffs, shared files and stale instructions. 2. **Give every step one owner.** Write it down in a table that agents read. 3. **Put the strong model where judgment is: planning and review.** Use the cheapest builder that passes review, and measure it. In one 41-item test, our cheapest builder missed fewer items than options that cost 16 to 20 times more. 4. **Give builders one item at a time.** Each item comes with its own short brief, so stale rules have less room to carry over. 5. **Re-test your structure when models change.** The team that was right in August was wrong in October. Want a team like this set up in your repository? Email [hello@jeffkent.dev](mailto:hello@jeffkent.dev). --- Source: https://jeffkent.dev/writing/subscriptions-vs-api/ # 235 billion tokens of agent work: subscriptions vs metered API **Topic:** AI cost · **Published:** October 2026 · **Reading time:** 7 min From February to October 2026, my AI coding agents used 235 billion tokens. At API list prices, that usage is worth about $124,000. The subscriptions that ran it cost about $4,000. That is about 31 times the value at list prices. This article shows how I measured it, where the value came from, and when a team should still pay for metered API access. ## The numbers | | February – October 2026 | | --- | --- | | Tokens | 235 billion (197 billion in 1.5 million Claude messages, 38 billion on Codex) | | Value at API list prices | about $124,000 (Claude about $103,600, Codex about $20,100) | | Subscriptions | about $4,000 (20 seat-months at $200: one Claude seat Feb – Sep, one Codex seat Aug – Sep, then nine employer-paid Claude seats and one Codex seat in October) | | Ratio | about 31× | | October 2026 only | $48,500 of usage on $2,000 of seats, about 24× | The usage covers my Claude and Codex agent work: a regulated crypto platform where I am CTO, a streaming platform rebuild, and my own video studio. The Claude agents ran in Claude Code and the Codex agents ran in the Codex CLI, each signed in to a flat-rate plan instead of an API key. ## How I measured it Both tools keep a local transcript of every session, with the token counts of each model response. I wrote a counter that reads every transcript on the three machines that ran agents, including the compressed archives. - **Duplicates.** Agents copied transcripts between machines, so the counter keeps each message once, by its message id. - **Prices.** Each message is priced at the published list price of its model: input, output, cache writes (at 1.25 or 2 times input, by cache duration) and cache reads. Where a provider does not list a cached-input price, I used 0.1 times input. - **Seats.** Until September I ran on one $200 Claude plan, plus one $200 Codex plan from August. In October the team moved to nine employer-paid Claude seats, with one Codex plan alongside. Each seat is priced at $200. The result is an estimate in both directions: - **The value is low.** Claude Code deletes transcripts after about 30 days by default. Much of the work before September is gone, so early months show only a fraction of the real use. - **The spend leaves some costs out.** Some Codex work used paid credits beyond the seat price, which this total does not include. DeepSeek ran on its own pay-per-token API and is excluded. I report "about 31×" for that reason, as an estimate, not a floor or a ceiling. ## Where the value came from **Context, not output.** About 97% of the Claude tokens were cache reads: the agent re-reading its instructions, the files it opened and the conversation so far, on every turn. Output tokens were about 0.3%. That share decides the economics. On a metered API, the cost of a long agent session grows with every turn because each turn re-reads the history, even at the cache price. On a flat-rate plan, the same re-reading counts against a usage window, not a bill. **The strongest model did the most work.** Claude Opus 5.5 alone was about $47,300 of the value, 38% of the total. It planned, reviewed and handled the changes that touch money, identity checks and sign-in. On a metered API, I would have pushed much of that work to cheaper models for cost alone, not for quality. ## What flat-rate plans cost in other ways A subscription is not free capacity. It moves the limit from money to time. - **Usage windows.** Each account has a five-hour window and a weekly limit. A guard holds new background agents when an account reaches 90% of its window, and lanes wait for the reset. - **Context still matters.** Because cache reads dominate, a bloated agent burns its window faster. Short briefs and small, one-item builder tasks keep each window productive. - **Cheap models still matter.** A small model as builder uses a fraction of the window that a large one uses. In one 41-item pilot, the small builder also had the fewest review misses. ## When to pay for the API instead Flat-rate plans fit agents that a person runs for their own engineering work, inside the provider's official tools and within the plan's limits. Use metered API access when: - **The model serves your customers.** A product feature needs its own key, rate limits and billing, not a person's seat. - **The workload runs unattended at scale.** Batch jobs and pipelines with no person behind them belong on the API. - **You need guarantees.** API plans give the service levels, data terms and spend controls that a team plan may not. Read the current terms of each plan before you build on it. Plans and limits change often. ## What I would tell a team 1. **Measure before you choose.** Run your agents for a month and price the transcripts. The ratio depends on how much your agents re-read. 2. **Keep transcripts longer than 30 days** if you want to measure later. I lost most of my early data to the default. 3. **Treat usage windows as a scheduling problem.** Schedule long jobs across hours, and stop new work before a window runs out. 4. **Spend the savings on quality.** Use the strong model for planning and review, where its judgment pays. Want help setting up an agent fleet that stays inside its budget? Email [hello@jeffkent.dev](mailto:hello@jeffkent.dev). --- Source: https://jeffkent.dev/writing/benchmarking-decision-models/ # Benchmarking decision models before trusting them **Topic:** LLM evals · **Published:** October 2026 · **Reading time:** 8 min In September 2026 I tested a small decision model as the judgment layer of an AI agent fleet. It decided which agent takes each request, how risky the request is, and whether to hold it for a check before any agent writes code. The headline result was strong: 16 of 16 correct, against 13 of 16 for a frontier model. The more useful results were the ones that went wrong first. This article covers the benchmarks, the mistakes in my own measurements, and the rules I now apply to any evaluation. ## The model Jev (TypeSafe) is a typed judgment model. It does not write text. It answers one question at a time from a fixed set of options (multiple choice or yes or no) and returns a confidence for its answer. I used it in three places: - **Classification:** what kind of request this is, and how risky. - **Risk checks:** yes-or-no questions such as "does this change send data to an outside service?" - **Routing:** which agent owns the request, and whether to hold it. ## Result 1: classification On 16 hand-labeled prompts, Jev labeled all 16 correctly. A frontier model labeled 13 correctly (0.813). | | Jev | Frontier model | | --- | --- | --- | | Correct | 16 of 16 | 13 of 16 | | Mean wait | 919 ms | 8,966 ms | | Tokens per case | 794 | 21,167 | Read the token figure with care. Most of the 26× gap came from the 20,000-token system prompt the frontier model carried in that setup, not from the task itself. Sixteen cases also make a small sample. The result showed that the small model was good enough for this job, not that it is better in general. ## Result 2: risk checks, and a holdout that changed the story I tuned the wording of each risk question on a set of known cases. The tuned questions agreed with my labels on 93% of them. That number was in-sample: I had chosen the wording on the same cases I reported. So I froze the wording and wrote a new holdout set after the freeze. Agreement dropped to 77–86%. Worse, it had one false negative that repeated 30 times out of 30: a change that sent a customer's email and postal address to an analytics vendor passed as safe. For a safety check, the gate is zero false negatives, not a high average. I added explicit wording for the missed case ("sends any data to an external service, vendor or off-site destination") and wrote a new, fresh holdout of 10 cases, because a holdout you have already used is no longer a holdout. The new wording caught every hazard in it. Each check cost about 1/600 of a mid-tier model call, with a mean wait of 345 ms against 15.7 s. ## Result 3: routing, quoted as a range The router picked the owner of each request. I ran it four times on the same frozen set of cases, with no change in between. It scored 74.3% to 76.3% (one earlier run reached 79.6%). The model is not deterministic, so I quote about 76% ± 3, never a single run. Inside that number: - **Owner choice** was correct on 33 of 33 scored cases. - **Unsafe permissions** were zero in every run. In every run, the router granted no forbidden write. - **Holds** were the weak part. Only 28.6–31.6% of holds were needed, so about two in three held requests waited for nothing. The zero unsafe permissions and the unneeded holds came from the same design: when unsure, hold. That is the right default for safety, and it has a cost in speed. About two weeks later the router was retired, and the measurement method stayed. ## Mistakes in my own measurements **A baseline I forgot.** One rule set had "zero under-classified cases across 403 real cases". That was true and meant nothing. The rule could only answer the top risk tier, so it could never under-classify. A constant answer of "top tier" matched 59% of all cases. The rule matched 53% of the cases it covered. Without the baseline, 53% looked like progress. **Evidence that raised confidence and lowered safety.** I added a keyword scan as extra evidence for the question "does this change have side effects that cannot be undone?" The model went from answering 2 of 403 cases to 265 of 403, and every one of the 265 answers was "no". A keyword scan cannot tell a change that mentions email from a change that sends email. More answers looked like progress, but all the new answers were in the unsafe direction. I reverted it. Measure the direction of a change against the safe default, not only how many cases get an answer. **A spelling defect that made a phantom result.** One label was spelled `verify_after` in the data and `verify-after` in the options. 164 of 384 labels could not match, and one question looked like 3% agreement. After the fix it was 71%. The lesson: fail the build when a label is not in its option set. **Small samples.** On an open-weights model, four runs of a puzzle suggested maximum reasoning effort beat low effort (3 of 4 against 2 of 4). Ten runs gave 6 of 10 at both. Four runs on a noisy task cannot tell 50% from 75%. ## Rules I now apply to any evaluation 1. **Report the trivial baseline beside every headline number.** If a constant answer passes your test, the test is not evidence. 2. **Freeze the setup, then write the holdout.** Tuning on the cases you report is in-sample. 3. **Use a fresh holdout after every fix.** A used holdout has lost its value. 4. **Set the gate by the cost of an error.** For a safety check, one false negative fails it, whatever the average. 5. **Quote non-deterministic results as a range,** over repeated runs on the same data. 6. **Check the direction, not only the rate.** More answers can mean more wrong answers. 7. **Validate labels in the build.** A typo can create or hide a result. Want evals like these for your own agents or LLM features? Email [hello@jeffkent.dev](mailto:hello@jeffkent.dev).