Beliefs, With Receipts

This year I put agents into our code review, our QA stage, our releases and our company chat, and measured what happened. This is what I now believe about where software development is going, and what made me believe it. Most of the numbers come from the posts I wrote along the way. Where I have no number yet, I say so.

Review moves to agents, and verifying becomes the job. The review agent I built went through 696 pull requests in ten weeks. In the QA stage, the agent sent one task in five back for fixes, the same rate as the people before it. My own work changed: I read fewer diffs and spend the time deciding on what the agents flagged. I made the longer case in The Verification Crisis, and the numbers are in Three Agents on One Devbox.

Tests become the contract. I do not think tests die. An agent needs something to check its work against that is not its own opinion, and a test suite is that thing. The failures I saw were tests that proved nothing. The QA agent noticed that CI did not run Jest at all, so the green check meant nothing. A fix once passed unit tests, type checks and lint and still fixed nothing, because the tests covered only the paths the fix touched. If an agent writes against your tests, they are part of the spec.

The first number is usually wrong. My first measurement of programmatic tool calling said 64% faster and 22% cheaper. The 64% came from my own bug: one shell command waited two minutes for input that never came. After the fix, sixty runs gave 9% cheaper and maybe 9% faster, with the time saving at the edge of what sixty runs can show. In a performance week, an agent reported a photo gallery opening in 0.84 seconds instead of 4.2. The real gallery had gone from 375 ms to 1,070. As I wrote in Too Good to Be True, an agent produces numbers faster than I can doubt them. More in Forty Tool Calls in One Program.

Most bugs will be “we asked for the wrong thing” bugs. The marketing agent and I measured organic Google traffic separately and got the same answer. Both answers were wrong. We had both read the same field, and it counts paid ad clicks as organic. A sentence in the agent’s own method note gave it away. The gallery number above was partly the same kind of mistake: the summary picked the wrong definition of “loaded”. When the agent and the person checking it start from the same wrong question, they agree perfectly.

Coding CLIs hand their place to GUIs, and the terminal goes to the agents. Claude Code, Codex CLI and the rest are how this started, and I do not think most people will keep working in them. In my own product the CLI is already an engine under the hood: a daemon starts Claude Code or Codex, and the person sees a chat, a plan and a pull request. Developers are the exception for now. The developer agents I put in our team chat barely got used, because developers already have an agent in their terminal. I think that is a phase. The terminal itself stays, with a new user. I still write command line tools every week, but for agents: one CLI wraps Jira, Bitbucket, Jenkins, Kubernetes and our monitoring so an agent does not hand-write API calls.

The chat is the new front door, and most of the people coming through it don’t write code. When I put one agent into Microsoft Teams, nobody installed anything. They already had Teams open. In three weeks, 38 people used it. In one week, three out of four messages came from private chats, most of them business teams: finance, listing operations, content, growth, HR. In the same three weeks, 21 tasks were opened from those conversations and 7 have reached a pull request so far. Most requests never become a task. They are questions that used to go to an analyst or sit in a backlog. The numbers are in One Agent for the Whole Company.

Nobody should work as a meat proxy. By meat proxy I mean a person whose job is to carry text between systems: the request from a meeting into a ticket, the ticket into an agent, the agent’s answer back to whoever asked. In the flow I built for our company chat, nobody carries it. Someone describes what they need in their own words, and the agent opens the task, writes the plan and posts it back. The worker agents do not report in free text. Each step ends with one line that names the commit, and the server reads it. The Jira task moves through its columns by itself. Between the request and the pull request, a person does two things: approves the plan and merges the result. Both are decisions. If most of your day is copying between windows, an agent can already do that part, and the work that is left is the decision at either end. See One Agent for the Whole Company.

The hard part is permissions and data, not code. The customer-service agent was told in its prompt to stop after ten searches. One answer took 193 tool calls anyway. The limit now lives in a small proxy, and the proxy says no. The company agent’s data access is the intersection of what the person may see and what the room may see, and it is checked in code before every call. The model never holds the grant, so it cannot be talked into a wider one. Getting this right took more of my time than anything about the model. See The Agents Joined the Chat.

Product, backend, frontend, QA and DevOps stop being separate jobs. Those lines were drawn around what one person could hold in their head. Agents already do the first pass of review, QA and production monitoring for us, and the layer-specific work goes with them. I think everyone becomes a developer, the way some AI labs already call everyone a member of technical staff: you build, and you own the product decision behind what you build.

Knowing only the code is no longer enough. When an agent writes the code, the scarce engineer is the one who knows what to build and why. Someone who knows the business and its numbers, and also knows the backend, the frontend and how the whole system behaves, sees the need before anyone writes it down and can tell when a result is wrong. The gallery number was caught that way: a change that was not about the gallery could not have made it five times faster, and you only know that if you know the system. That person is the one I cannot hire enough of. An engineer who knows only the code is now doing what the agent does.

The software factory is the new line of work. The harness around the agents, meaning its permissions, tools, checks and metrics, is becoming a product of its own, and someone has to run it. Business teams are already inside ours. They ask for analyses, reports and changes, and they will not read the diff. When an agent does a job for them, the factory is what has to catch the mistake. So developers become the people who watch it, read its metrics and fix it. Most of what I fixed this year was in the factory. A shell command that waited for input skewed my first measurement. My own product showed every run as free and hid the cost. An agent ignored its search limit until a proxy enforced it. When someone in our company chat gets refused, they ask me, and I add a line to a file. Most of my own time with the agents goes into this.

Taste stays human, and it is the 1%. I have no number for this, so read the 1% as a way of saying it is small. An agent will write the faster algorithm when asked. What it did not do in that performance week was look at the gallery number and ask where it came from. It did not open a filter drawer on a real phone and see the empty rows that four headless runs had passed. Once, after a twenty-minute measurement, the right answer was to leave the code alone. Knowing what is wrong before the numbers show it, and what not to build at all, is the small part of the work where people still matter most.

Big teams become small teams with agents. Earlier this year I wrote the 1-Person Unicorn Playbook about founders. Inside a company, I think the unit is a small team. In four days, one person and an agent produced nine pre-production releases. My part was merging every pull request, deciding what went into each release and testing on my own phone. Those human steps are where the work waits. In the 30 days before October 25th, 13 to 15 of the 22 pieces of work the company agent was coordinating were waiting on a person. One person becomes that bottleneck. A few people who share the approvals, the tests and the decisions do not.

Route work by judgment, and keep jobs short. I do not think the cheapest model wins, and I do not think the strongest model should do everything. Review, root-cause analysis and implementation go to the strongest model. Collecting and summarising data that is already there goes to a cheaper one. When I counted the agent runs on October 31st, 129 runs that went past 300 turns had used 35% of the tokens spent on the strongest model. What fixed it was splitting the work into smaller jobs with a written handoff between them.

Token cost is an engineering number. I now look at cost per run while designing, not at the end of the month. In my tool-calling work, 86 to 93% of every request was read from the prompt cache, so one wrong cache price changed every dollar figure in the study. For a while my own product showed every run as free. For the next few years, how many tokens a team can get will decide how fast it can build.

Good code means cheap context. No number for this one either. Line width and blank lines matter less as fewer people read each line. Small modules, clear names and less code still matter, because every line an agent reads is context it pays for and a place it can go wrong. Good code used to mean easy for a person. It is starting to mean cheap for an agent.

I wrote these down so I can be wrong about them in public. None of this will happen quickly. Old software will run for a long time, and people will still be paid to write code by hand in ten years. In November 2027 I will come back to this list and mark what held.