What Ten Months of Git History Says About AI Coding Agent Productivity

Introduction

Claims about AI coding agent productivity usually come from vendor benchmarks or from developers reporting how fast the work felt, which is why we keep returning to the question of which AI coding tools actually hold up in production. We build health software for a living, where a defect reaches a patient rather than a dashboard, so we wanted the answer from an artifact that cannot be talked up after the fact: merged pull request history on a production Android codebase several years old, plus a session-by-session record of how the agent actually performed while writing it.

Before any number below, the limit on all of them. This covers two engineers on one project, compared across a calendar boundary rather than in a controlled trial, and the second window carried a heavier feature roadmap planned independently of any tooling change. Output rose, and the product owner confirmed independently that features which used to slip were landing on schedule. How much of that belongs to the agent, as opposed to what happened to be scheduled, is not something this data can settle. The baseline needs one clarification too. The earlier window was not manual development: both engineers were already writing code with an AI assistant in the editor. What follows compares an agent-led workflow against AI-assisted typing, so the first jump, from no assistance to assistance, sits underneath the baseline rather than inside the gain. The project measured here is not itself a regulated health product, which is worth saying plainly rather than implying otherwise: the delivery numbers come from a commercial Android codebase, and the parts that carry over to regulated work are the workflow and the controls, not the percentages.

What follows is the delivery data, then the session data underneath it, then the workflow that produced both, and finally why a feature can still take real time to ship when an agent writes essentially all of the code. If you build health applications, read the last two sections first. Throughput is the least interesting thing an agent changes about a regulated codebase; what matters is where the human checkpoint sits, what a reviewer stays accountable for, and what none of it changes about the controls a regulated codebase already carries.

What the Delivery Data Shows About AI Coding Agent Productivity

Both comparison windows are five full calendar months, so raw counts compare directly. Adoption was staggered, with the second engineer starting about a month later, so their post-adoption window reflects four months of real use.

The size figure is worth one sentence of explanation. The median nearly doubled while the mean held flat at around 375 lines, which means the whole middle of the distribution moved up rather than a few large changes stretching the average.

Splitting every pull request into four streams shows where the capacity went. New features rose from 55 units to 85. Platform and infrastructure work went from 12 to 22, the fastest-growing category by percentage. Fixes to older code fell from 35 to 17. Because units are not the same size, each was also scored from 1 to 5 for breadth and complexity, and those scores summed into a weighted figure.

Adding the first three streams and excluding rework gives the headline: total new-value output, complexity-weighted, rose about 40%. It lands at +42% for one engineer and +39% for the other, which is why we quote it ahead of the +55% feature count.

The Rework Number That Went the Wrong Way

Fixes an engineer makes to their own recently-shipped features are the closest available proxy for whether speed is being borrowed against future work. The data refuses to be tidy here.

The figure is a count, not a defect rate. It is the average number of fix pull requests that landed against each feature pull request, so it says how often a feature needed a follow-up, not how many bugs reached users.

One engineer's rework rate doubled. The two started nearly four times apart on this measure and finished within a quarter of a point of each other, so how much correction a feature needs now depends far less on who built it. The team average improved.

The warning sits in the absolute numbers rather than the ratio. Both engineers are shipping considerably more features, so review and QA carry a heavier load even where the per-feature ratio looks fine. If throughput climbs and review capacity does not, the constraint moves rather than disappearing.

What This Analysis Cannot Tell You

It is observational, with a sample of two engineers who hold different roles and are assigned different kinds of work. It also says nothing about the move from manual coding to AI assistance, because that step had already been taken before the earlier window opens. And it describes one project on one platform. How much an agent helps depends on how much of the work happens to be the kind an agent does well, which varies with the stack, the domain and how much of the codebase is already written down, so a delivery gain measured on a mature Android product is a figure that will shift with those variables, not one to expect to hold as-is on a different stack or domain. There is no quality or defect measure anywhere in it: rework is a proxy built from which files a fix touches, and whether more bugs reached users is outside what git can answer. The complexity score is an ordinal judgement read from pull request titles and commit messages, which makes it the softest column in the whole analysis.

The defensible claim is narrow. Feature output rose 55% to 70% and total new-value output rose about 40% alongside adoption, with no offsetting rise in rework per feature.

How Often the Agent Gets It Right First Time

Throughput answers what shipped. The more useful question is what the agent is doing well, what it still gets wrong, and how the difference gets caught. That comes from a separate record of roughly 89 AI-assisted sessions over about 16 days, graded on outcome and split into an earlier and a later period.

Raw grading looked similar across both periods, with around 90% of sessions marked fully or mostly achieved either way. "Mostly achieved" turned out to be carrying a lot of weight in that sentence. Re-reading the later period's sessions individually, roughly two thirds had no recorded problem at all: right files, right approach, clean multi-file edits, marked down only because nobody typed an explicit confirmation into the transcript. The remaining third needed exactly one small correction, a single redirect such as "that is over-engineered, simplify it" or one fix to make something compile. None needed a from-scratch redo.

Re-scored on that basis, counting only sessions where the agent actually produced code:

Separately, about 11% of later sessions were blocked before any code existed, by expired logins, bad dependency versions, and similar environment problems. Those are excluded from the table because "was the code right" does not apply to a session that never reached code.

One result is worth sitting with. Hitting a wrong turn along the way usually did not mean the session failed. Sessions that hit a rejected suggestion, a compile error, or a misread instruction mostly still finished successfully once redirected. A high success rate here is not the product of the model rarely making a wrong move. It is the product of wrong moves being caught early and undone cheaply, which suggests the metric worth optimising is the length of the loop between "something is off" and "fixed", not the model's error rate.

Written Conventions Eliminate Whole Classes of Error

The strongest single piece of evidence in the whole exercise came from a comparison nobody designed. Same engineer, same period, two codebases: one with years of documented conventions behind it, one started from scratch.

The only material difference was whether the conventions existed yet. The new codebase ran on the strongest model available, with access to the mature codebase and the same working rules, and the error rate was still double. It came down only once the first few implementations were carried over and the new codebase accumulated its own written history.

Access to a bigger model did not compensate for context that was not written down in the right place. That result shapes what comes next: the practical lever is not model selection, it is what the repository tells the agent before it starts.

Where the Agent Is Strong, and Where It Is Not

The pattern generalises, and it matches what we see when AI agents run inside a healthcare product, where the checkable part of a task is the part that can safely be delegated. Agents are at their best on closed-ended problems, the kind they can check for themselves as they go. Does it compile, does the test pass, does the type check succeed. Give a clear target and an automatic way to tell right from wrong, and the model iterates on its own until it gets there.

Open-ended problems have no such check, because the real question is whether the result is what the business needs and nothing in the code can answer that. Two failure patterns follow. The model can optimise hard for the job exactly as written and skip something a person would have caught by instinct. And it can miss context that was never written anywhere it could read: an overlap with an existing feature nobody re-described in the ticket, a UX behaviour implied by a design file but never stated, or a technical detail that quietly changes what the business logic should do.

The failures we logged fit that split closely, and almost all of them fell into two groups. Either a convention already existed nearby in the codebase and was not applied consistently, or a judgement call was made without surfacing the trade-off it involved. One was a plain language-level rule violation that did not compile, a reminder that a strong one-shot rate is not a perfect one.

None of those look like an agent producing nonsense. The code ran, compiled, and looked reasonable at a glance, which is exactly why a careful read-through catches them and a passing test suite does not.

The Workflow Underneath the Numbers

Building a feature is not one long agent session. It runs as four steps, each producing its own committed document before the next one starts, and each reading the previous step's document plus whatever project rules are active.

Brainstorming explores the idea and writes down what problem this actually solves. UI analysis reads those notes and the design file, then records the screens, states, and interactions involved, including behaviour a designer implied but never spelled out. Planning reads both and turns them into the document a person and the agent both agree to before any code exists. Implementation builds from that plan, which makes the step most people picture when they hear "the AI wrote the code" the last of four.

The plan is the important artifact, because it does the job a specification is supposed to do. A good one names what is being built in plain language, breaks the work into steps that each carry their own confidence rating, lists exactly which files will be modified, created, or need a careful look, sources its test cases from the project's own documented conventions rather than inventing them, and states explicitly what is being left for later so future scope does not creep into what ships now.

Chaining documents this way is what makes a failure diagnosable. At each step the agent is largely limited to what that step's document contains, so instead of asking where in one long opaque run the plot was lost, you check one bounded document and find which step was short, wrong, or misread. When something ships broken, the first question is not what the model did wrong. It is which document was missing something.

That question has four possible answers, and only one of them gets better as models improve. The model failed despite clear intent and sufficient context. The context was insufficient, so the output looks correct but violates a constraint nobody wrote down, and the fix belongs in the rules file rather than in the code. The intent was misunderstood, because the instruction was ambiguous. Or the intent itself was wrong, and seeing the thing built is what revealed that the requirement should change. Recurring corrections get written into the project rules, often in the same commit as the fix, and the error categories that got codified stopped coming back.

Why a Feature Still Takes Real Time

An agent writes essentially all of the code in this workflow, and a feature still takes real time to ship. Those two facts stop looking contradictory once you see where the time goes.

The old work was a chain: understand the problem, decide an approach, write the code, check the result. Agents mostly remove one link. The rest is still there, still human, and does not speed up because the middle step did. Time now goes to analysis and to finding the business rules that were never in the ticket, to compressing a fuzzy goal into a specification precise enough to generate from, to reviewing code you did not write yourself, which is slower per line than reviewing your own because writing something builds your mental model of it for free, to the correction loop that runs at the pace of human attention rather than model speed, and to the judgement calls that stay serial no matter how much compute is available.

Edge cases used to surface during implementation, discovered while writing the thing. They now have to be found before the spec is written, which moves the same work earlier rather than removing it.

Some of what a product should do cannot be written down in advance, because nobody knows to write it until they see the built result violating it. A half-filled form when the user backs out and returns. A record edited on the phone and changed on the server at the same time. An approval landing while someone still has the item open for editing. A screen restored from the background after the system killed the process. Acceptance criteria that all pass while the feature is still wrong, because the code already contained a truth the ticket's author did not know about. No rules file catches these. A person who understands the product looking at the finished result does.

Android sharpens every one of these. Iteration is expensive, since seeing a change means a build and a manual walkthrough rather than a near-instant refresh. Work is serial by nature, one device and one app state at a time. Mobile sits at the end of the requirements chain, so business rules arrive as tickets that have already lost detail. Business logic lives in the combinations of what a screen can be doing at once, which exist in full in exactly one place, and shipping a mistake is expensive because releases go out in batches and roll out gradually.

Review Is Where the Judgement Still Lives

Code an agent writes goes through the same review as code a person writes, held to the same standard, before it ships. Nothing about the tooling lowers that bar, and it arguably raises the value of a careful reviewer, because the reviewer is now checking work they did not personally write.

We stopped double-checking a compiler's output because compilers are exact, fully specified, and have been proven reliable for decades. A language model cannot earn that same trust by its nature. Review depth can scale sensibly with how novel the change is and how much damage it could do, but it does not disappear.

The fault categories are the reason. Most of what a careful review catches is not the model getting the coding wrong. It is missing context, a misread instruction, or a requirement that was wrong to begin with, and a better model fixes none of those. Only a person looking at the result does. A plan is intent captured before the work exists, so turning it into thousands of lines of code means resolving hundreds of small unstated decisions along the way, and the finished code is the only place those decisions become visible. Checking the result against the plan verifies the decisions you already made and says nothing about the ones the model made for you.

Two things follow for anyone applying this in a regulated product. The human checkpoint belongs where errors are cheapest, at the plan and before the code. And the controls that already govern a codebase apply unchanged to agent-authored code, with no exemption for the fact that a model drafted it. That is how our own regulated work is governed, under the same HIPAA development controls and ISO 13485 design controls we would hold a change to if a person had written every line. Tooling changes how a change gets drafted, not what it has to satisfy before it ships.

Measuring AI Coding Agent Productivity on Your Own Codebase

The delivery method transfers. Find the adoption date in the repository itself rather than asking anyone to remember it, and expect one date per person rather than one for the project. We used two: the commit that introduced the agent's configuration file for the first engineer, and the first specification document committed by the second, which is the earliest point their work is certain to have run through the agent. Pull every merged pull request in a defined window and label each one Feature, Infra, Fix, or Other. Count lines with specification files excluded, since a generated plan document inflates a diff without adding product code. Score each unit from 1 to 5 for breadth and complexity, read from the pull request title and its commit messages. Separate rework from maintenance with a diff-based lookback: a fix that touches code from a feature shipped in the previous 60 days counts as rework against that feature, and anything older is maintenance and stays out. Match the before and after windows in length so the counts compare directly.

The session method transfers too, and is cheaper to start. Grade each session on outcome, flag whether it hit a wrong turn, and re-read the middling grades individually rather than trusting the label. That is where the interesting result was hiding here.

Then publish the caveats alongside the numbers. If you want the infrastructure and security side of AI-assisted delivery answered properly, alongside how we approach building secure AI models for healthtech, talk to our engineering team.