The problem
The company needed its whole platform rebuilt: 9 apps, including customer sign-up, identity checks, crypto buy and sell, deposits and withdrawals, and the staff back office. The work touched money, identity verification and sign-in, where a careless change becomes a regulatory problem, not just a bug.
My goal was to make AI agents do most of the building, while keeping the review discipline a regulated company needs.
My role
I set the architecture and the acceptance criteria, and agents wrote most of the code under my review gate. My own architecture decisions: a NestJS modular monolith with 14 domain modules on PostgreSQL, Redis and Cognito; typed API contracts (ts-rest and zod) shared by the back end and 5 React web apps; money held in integer centavos until display; role separation and break-glass access on the money path, with two-person approval for compliance, fee and rate changes; a separate rate-polling worker per price source; and Terraform on AWS ECS, reachable only through Cloudflare. Every production deploy and every business-logic change needs my approval. Deterministic checks (tests, type checks, builds, scans) run automatically.
What I built
One job per agent
Each agent gets a single role: plan, build, review or test. A planner splits each ticket into small, clear items. A builder takes one item with a short, written brief: the files it may touch, the goal, and the checks that prove it is done. A builder never reviews its own work.
At peak, more than 45 agents worked at the same time across Claude, Codex and Grok, as long-running lanes plus short-lived helpers.
Model routing by risk and cost
I benchmarked models and settings on real tasks for cost and quality, then routed each job to the cheapest option that met the quality bar:
- Small model, default effort: mechanical, tightly scoped work such as one-file fixes, fixed checklists and scripted releases. Raising its effort to maximum cost far more with no gain in quality.
- Mid-tier model: normal feature builds and repairs, and items the small model failed twice.
- Strongest model, high effort: planning, every independent review, and anything that touches money, identity checks, sign-in, security, migrations or many files at once.
The result was about a 20-fold lower cost per coding task, with no rise in review failures.
Measuring decision models before trusting them
I tested Jev (TypeSafe), a typed judgment model that answers multiple-choice and yes-or-no questions, as a decision layer: which agent takes a request, how risky it is, and whether to hold it for a check.
- Classification: it labeled 16 of 16 hand-labeled prompts correctly, against 13 of 16 for a frontier model, with 26× fewer tokens (mostly because the comparison model carried a 20,000-token system prompt) and about a tenth of the wait.
- Risk checks: wording tuned on known cases scored 93% agreement, but a frozen holdout dropped to 77–86% with one repeated miss. A rewritten version then passed a fresh 10-case holdout with no misses, at about 1/600 the cost of a mid-tier model per check.
- Routing: 76% ± 3 accuracy over four runs, with correct owner assignment on 33 of 33 scored cases.
I retired it after about two weeks, after measuring that about two in three of its holds were unneeded. What stayed is the method: freeze a holdout, report the trivial baseline, and repeat runs before you believe a gain.
Open-weights models as workers
I ran open-weights models next to the commercial ones. DeepSeek V4.1 Flash did bounded build, rewrite and review work through Ollama and the DeepSeek API, including as the model behind Claude Code, with about six workers in parallel under the host’s concurrency cap. GLM 5.3 Flash handled scoping and build steps in a separate workflow. In a ten-run test, DeepSeek’s maximum reasoning effort solved a test puzzle no more often than its lowest setting (6 of 10 at both), although a four-run sample had suggested otherwise.
Subscriptions, not metered API
I ran the Claude and Codex agents on flat-rate subscriptions, not pay-per-token API keys. From February to October 2026, across my Claude and Codex agent work (this platform included), the agents used 235 billion tokens (197 billion in 1.5 million Claude messages, 38 billion on Codex). At API list prices that is about $124,000, against about $4,000 in subscriptions (one personal Claude plan until September, one Codex plan from August, then nine employer-paid Claude seats in October), so about 31 times the value at list prices. In October alone, $48,500 of usage ran on $2,000 of seats. About 97% of the Claude tokens were cache reads of earlier context, which is why trimming each agent’s context mattered.
Independent review on every change
Every change needs an approval from a separate reviewer agent, made against the exact version being merged, on top of the automated gates. Every release to customers also needs a human approval. Reviews of user-facing work include screenshots and a checklist of small details (wording, spacing, empty, loading and error states), because many small misses add up to a poor product. Approved changes land in batches, and a staging build is verified after each batch.
Code quality the build enforces, not reviewers
AI agents copy code readily, so a fast agent team can produce a large, repetitive code base. For the rewrite of the staff back office front end, I made reuse a rule that the build checks:
- One shared component kit (58 parts so far: tables, drawers, dialogs, status chips, empty and error states). A missing pattern goes into the kit first, with its own tests, before any page uses it.
- Pages cannot add their own styles. Colors, spacing and type sizes are tokens defined once, and status and error wording is defined once per value.
- Any copied block of 50+ tokens, any unused file or export, and any file over 200 lines fails the build.
The app it replaces, area by area, had a single file of 7,000 lines. A reviewer still requests changes for copied markup or a one-off version of a kit part, but the build catches most of it first.
Safety boundaries
Agents cannot delete files, send email, or change the live system. Every release to customers needs a human approval, and the staging and production environments are always named in every report, so a staging result can never be mistaken for a production one. A test that runs against mocks is never accepted as production evidence.
Shared memory that survives sessions and providers
Agents forget everything between sessions, and different providers’ tools do not share state. I built a shared memory of 600+ short notes: rules, decisions and lessons from past incidents, each with the reason behind it. Agents read the relevant note before they act, and add one after a verified outcome.
Delivery infrastructure
A scheduler spreads builds and tests across three machines (196 jobs in one day) and caps heavy jobs so agents do not starve each other. The test suite includes 1,000+ Playwright end-to-end tests, and a Telegram bot connects 33 team chats to the agents so staff can raise requests where they already work.
Results
- About 4,000 approved code changes in under six months across 9 apps (TypeScript, NestJS, React, PostgreSQL, Terraform on AWS).
- 3,000+ automated test files, and an independent approval on every change.
- A back office front end rebuilt on 58 shared components, where copied code, unused code and oversized files fail the build.
- About 20-fold lower cost per coding task after routing work by risk and cost.
- About $124,000 of AI usage at API list prices on about $4,000 of flat-rate subscriptions (personal and employer-paid), February to October 2026, across all my agent work.
- The platform replaces the company’s current platform and is in final testing.
The same approach let me rebuild a streaming service as its only engineer: a viewer website, a staff back office with its back end, and native iOS and Android apps with 600+ tests, now in testing.
Lessons learned
- Fewer, stronger agents beat more agents. At peak, 45+ agents ran at once, with separate lanes for features, bugs, backlog, review and chat. As models improved, a strong planner needed far shorter instructions, so I closed most of those lanes. Today 10+ long-running lanes do the work: one planner splits each ticket into small items for low-cost builders, and independent reviewers check every merge.
- Report the trivial baseline. A risk-tier rule matched the recorded tier on 53% of the cases it covered, while always answering the top tier matched 59% of all cases. Without the baseline, 53% looked like progress.
- Context is the main cost. Agents that reload long histories spend most of their budget re-reading. Each agent now starts with only the brief it needs.
- Agents must fail loudly. An agent waiting on a prompt nobody sees blocks its queue in silence. Agents now stop and report a blocker instead.
- Set scope and effort per task. Unscoped agents wander, and maximum effort on small tasks costs more without better results. Every brief names its files, its goal and its stop point.
- Every helper needs an end condition. Tunnels, dev servers and watchers outlive the agents that start them unless each job ends with a cleanup check.
What I bring to your team
- A working model for AI-assisted delivery that keeps review, security and release controls intact.
- Measured cost and quality decisions, not guesses about which model to use.
- Hands-on engineering across TypeScript back ends, React, native mobile, AWS and Terraform, with deep experience in regulated money and identity flows.
- Each client’s code stays in that client’s own repositories.
