The comprehension gap: 200 engineers, 12 months, and what agentic coding did to recall

The question we set out to answer in early 2025 was narrow and awkward: the throughput increase from agentic coding tools is real, so what does it cost? Most available evidence was self-reported, vendor-funded, or measured over a sprint or two. None of it followed the same engineers long enough to see whether the capability underneath the output was still there.

So we followed 200 of them for a year. This report is the full method and result set. It contains one finding that is good for our product and one that is not, and we have tried hard not to let the first one do the arguing for the second.

Cohort and setting

Participants were recruited from 14 companies ranging from a 12-person startup to a 2,000-person public company, across fintech, developer tooling, logistics and health records. Every participant was a working engineer shipping to production in a repository owned by their team. Nobody was paid per assessment, which matters: paying for assessment performance buys you assessment performance.

The seniority split was deliberate and turned out to carry most of the interesting variance. We over-recruited juniors relative to the industry distribution because the hypothesis we cared about, that assistance substitutes for the practice that builds capability, predicts the largest effect where there is least capability to begin with.

What we measured, and how

Throughput used a composite: tickets closed per sprint divided by the team's own complexity estimate, multiplied by PR merge rate, adjusted for team size and codebase age. It is not a perfect metric. It is the metric these teams already argued about, which made it auditable by the participants themselves.

Comprehension is the harder half. Each month we selected a diff the participant had merged 30 days earlier and ran a structured interview against a five-point rubric: can they explain what the code does, reproduce the control flow from memory, identify the failure modes, make a targeted modification, and predict the effect of a change we proposed. Two raters scored independently. Inter-rater agreement was 0.81 (Cohen's kappa), which is acceptable but not excellent, and is the weakest link in this study.

Throughput went up, unevenly

Across all seniority levels, daily tool users showed a statistically significant throughput improvement against their own pre-study baseline. The mean was +34%, with a wide distribution: the bottom quartile saw almost no change, the top quartile nearly doubled.

The gain was inversely related to experience. Senior engineers improved 22% on average, juniors 47%. Our reading is that senior engineers were already spending most of their time on work that generating code does not shorten: deciding what to build, negotiating an interface, working out why a system fails under load. Juniors were spending more of their time on the part an agent is genuinely good at.

Comprehension went down, and kept going

At 30 days, daily users scored a mean of 2.6 of 5 on AI-authored code against 4.1 on code they had written unassisted. That is the 37% gap. At 90 days the gap widened rather than closing, which rules out the most comfortable explanation available to us, that comprehension is simply delayed and arrives with familiarity.

The blunt version of the 90-day number: participants could not reliably explain or modify close to half the code in repositories they owned. Code with their name on the commit, in a system they were on call for.

The effect scaled with usage intensity, and not linearly. Engineers who routed more than 60% of their working code through an agent scored 0.8 points below those in the 30 to 40% band. Below roughly 40% we could not distinguish the groups. There appears to be a threshold rather than a smooth dose response, though we are wary of reading too much into an inflection found in a single cohort.

The two measures pull against each other

High throughput and high comprehension were not positively correlated, which is the result we expected to find and did not. They were weakly negatively correlated (r = -0.18, p < 0.01). The engineers with the largest throughput gains tended to show the largest comprehension deficits.

A weak correlation is still worth stating carefully. This is not a law, and it is not destiny for any individual engineer. It says that in this cohort, the two things an engineering organisation wants from a tool were mildly in tension, and that measuring only the first will systematically mislead you about the second.

Seniors were largely insulated. They scored 3.8 even on AI-authored code, below their own unassisted baseline of 4.4 but far above the junior figure of 2.1. The most plausible mechanism is that they read generated code against an existing model of the system, so review is itself an act of comprehension. A junior has no such model yet, and reading fluent code does not build one.

The intervention: one day a week without the agent

In the final three months we asked 60 participants to designate one working day per week as agent-free: editor, documentation, and their own head. We tracked throughput on those days separately from assisted days.

Unassisted throughput was 38% below assisted throughput, exactly as you would expect and not the point. The point was the trend. In month one, agent-free days averaged 2.1 tasks per day. By month three they averaged 2.6, a 24% improvement in unassisted capability over twelve weeks. A matched control group of engineers who did not do agent-free days improved 4% on the same measure over the same period (p = 0.003).

Total delivery did not fall. The agents were still present four days in five, and the assisted days absorbed the difference. Deliberate practice without the tool raised the floor without lowering the ceiling. That is the single most useful thing this study found, and it is the finding the product is built around.

Limits, and what would change our minds

• The 30-day comprehension window is probably too short. A 60 or 90 day primary window would give a cleaner picture, at considerable cost in attrition.
• The comprehension rubric is ours and has not been externally validated. We are working with two university groups to fix this and will publish the revision whichever way it goes.
• Inter-rater agreement of 0.81 is adequate, not strong. Some of the reported gap is rater variance.
• Participants knew they were being assessed monthly, which plausibly raised comprehension scores across the board. If so, the real gap is wider than we report, not narrower.
• We cannot separate tool use from the kind of work that invites tool use. An engineer assigned mostly boilerplate will both use the agent more and retain less, for reasons that have nothing to do with causation.
• Fourteen companies is not a representative sample of the industry, and none of them were in defence, embedded, or scientific computing.

The result that would most change our position is a replication with an externally validated instrument showing no widening between 30 and 90 days. That would move the story from capability loss to delayed familiarity, and most of what we build would need rethinking.

Pre-registration, rubric, rater instructions, the anonymised score matrix and the analysis notebooks are published in full. The raw interview transcripts are not, because they contain proprietary code discussed in detail under NDA.

Related work

Peng and colleagues [1] established the throughput effect for AI pair programming under controlled task conditions, and our assisted-throughput result is consistent with theirs at a larger scale and over a far longer window. The comprehension half of this study owes its method to the retrieval-practice literature, principally Roediger and Karpicke [2], and to Bjork's work on desirable difficulty [3], which predicts that assistance removing effort will also remove learning. Sweller [4] provides the load framing we use to explain the seniority interaction. On the measurement side, Forsgren and colleagues [5] argue that delivery metrics must be paired with a counterweight, which is precisely the failure mode we document when throughput is reported alone. Ziegler and colleagues [6] report acceptance-rate telemetry at scale; our AWL-2026-04 result on the predictive value of that metric follows directly from this cohort.

Artefacts

Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result.

References

1. Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590.
2. Roediger, H. L., & Karpicke, J. D. (2006). Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention. Psychological Science, 17(3).
3. Bjork, R. A., & Bjork, E. L. (2011). Making Things Hard on Yourself, But in a Good Way. In Psychology and the Real World.
4. Sweller, J. (1988). Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science, 12(2).
5. Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution Press.
6. Ziegler, A., et al. (2022). Productivity Assessment of Neural Code Completion. MAPS at PLDI.
7. Bhardwaj, M. (2026). Lines accepted predicts nothing: six productivity proxies against six-month capability. AI Work Lab Research, AWL-2026-04.