Lines accepted predicts nothing: six productivity proxies against six-month capability

If you buy an AI coding tool for an engineering organisation, you will be shown acceptance rate. It is easy to instrument, it goes up when the tool is working, and it is almost entirely uninformative about whether your engineers are getting better or worse.

This paper is the quantitative version of an argument we have made in prose. We had the unusual luxury of a cohort with both instrumented tool telemetry and six-month capability assessments, so we could ask directly which of the commonly-reported numbers predicts the thing everyone claims to care about.

The six proxies

We selected proxies that appear on shipping dashboards from major vendors, including our own. Every one of these is a number some engineering leader is currently reading in a slide deck.

The outcome variable is the comprehension score described in AWL-2026-06, measured six months after the telemetry window. 168 of the 200 participants had complete records on both sides.

Results

Acceptance rate sits at r = 0.04. That is not a weak signal, it is the absence of one. Lines accepted is worse than useless at r = -0.11, weakly negative: the engineers merging the most agent-authored code were, if anything, slightly less capable six months on.

Modification rate is the standout at r = 0.51. An engineer who accepts a suggestion and then changes it has read it, evaluated it against the system, and formed an opinion. That is the observable trace of comprehension happening. An engineer who accepts verbatim may have done the same thing silently, but on average, they have not.

Combining them helps, and still is not enough

A three-signal composite of modification rate, explain-check pass rate and review participation reaches r = 0.63 against six-month capability. That is genuinely useful for spotting a team-level trend, and we ship it.

It is not enough to make a decision about a person. An r of 0.63 explains under 40% of variance. Used on an individual, it would misrank a substantial fraction of any team you pointed it at, and the engineers most likely to be misranked are the ones working on the least typical code. We say this in the product, next to the number.

The gaming problem is not hypothetical

During the study one team learned that modification rate appeared on a dashboard their director read. Within three weeks their modification rate rose 22% with no corresponding change in comprehension scores. Engineers were making cosmetic edits to accepted suggestions, in one case a script that reformatted whitespace.

We report this because it is the strongest argument against our own recommendation. Modification rate is the best proxy we found and it collapsed inside a month once it became a target. Any metric in this space should be treated as a smoke detector rather than a thermometer, and should never be attached to an individual's review.

Limits

• Single cohort, and the same cohort as AWL-2026-06, so the two papers share every sampling weakness.
• Telemetry came from four different tools with different definitions of acceptance. We normalised as best we could and the residual noise favours our conclusion, which we do not like.
• The capability measure is the unvalidated rubric discussed at length in AWL-2026-06.
• Correlation with a six-month outcome is not evidence that intervening on the proxy changes the outcome. Nothing here licenses a target.
• The gaming incident was observational and involved one team of nine.

Normalisation code, the composite definition, and per-proxy scatter plots are published. Vendor-attributed telemetry is aggregated before release under the terms we agreed with participating companies.

Related work

Ziegler and colleagues [1] report acceptance-rate telemetry at scale and treat it as a productivity signal; this paper is a direct test of that treatment against a capability outcome. The gaming episode we describe is the mechanism Goodhart [2] named and Ridgway [3] formalised for measurement in practice. Forsgren and colleagues [4] argue for paired metrics with a built-in counterweight, which is the structure of the composite we ship. On why acceptance without modification may indicate disengagement rather than quality, Bainbridge's account of automation-induced skill decay [5] remains the clearest statement, and Parasuraman and Riley [6] give the misuse taxonomy we borrow. The capability outcome and cohort are documented in AWL-2026-06 [7].

Artefacts

Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result.

References

1. Ziegler, A., et al. (2022). Productivity Assessment of Neural Code Completion. MAPS at PLDI.
2. Goodhart, C. A. E. (1975). Problems of Monetary Management: The UK Experience. Papers in Monetary Economics, Reserve Bank of Australia.
3. Ridgway, V. F. (1956). Dysfunctional Consequences of Performance Measurements. Administrative Science Quarterly, 1(2).
4. Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution Press.
5. Bainbridge, L. (1983). Ironies of Automation. Automatica, 19(6).
6. Parasuraman, R., & Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39(2).
7. Bhardwaj, M. (2026). The comprehension gap: 200 engineers, 12 months, and what agentic coding did to recall. AI Work Lab Research, AWL-2026-06.