Designing a twelve-month cohort study of agentic coding: power, confounding and attrition by simulation

Two years after ChatGPT, the evidence on productivity is accumulating quickly and the evidence on capability barely at all. Controlled tasks show engineers finishing faster with assistance; field experiments this year show throughput gains across thousands of developers; a randomised study in a school setting found that unrestricted access to a model improved practice performance and harmed later unassisted performance. Nobody has followed working engineers long enough to see what happens to their understanding of the code they own.

That is the study we begin in January: working engineers, followed for twelve months, with monthly comprehension assessments on diffs they personally merged, and a nested intervention. This paper is not a result. It is the design analysis we did first, by simulating the study under the threats we could anticipate, and it is published before recruitment so that the choices it justifies cannot later be adjusted to fit the data.

The planned study

The primary analysis model is a linear model with engineer fixed effects. For engineer i in company c, assessed in month t on a diff with agent indicator A_it:

where α_i absorbs everything stable about the engineer (seniority, ability, team) and w_it holds diff-level covariates. Clustering by company, with a t distribution on G − 1 degrees of freedom, guards against company-level correlation in the effect, which with a dozen or so companies is the dominant inferential risk.

The simulated study

Each simulated engineer has a stable comprehension level (SD 0.6 rubric points) and a personal agent-usage share drawn from a Beta(2.2, 2.0) distribution that rises slowly over the year. Each month a diff is drawn with a complexity x_it ~ N(0, 1) that lowers comprehension by 0.35 points per SD. The engineer’s effect β_i varies around the cohort mean by 0.10 (0.15 in the attrition analysis), plus a company-level deviation where stated. Diff-to-diff noise has SD 0.75, and the two-rater average adds rater error set by the single-rater reliability:

Each design point was simulated 200 to 400 times. Unless stated, the design is 200 engineers in 14 companies, single-rater reliability 0.8, and 10% dropout at random.

Result 1: how many engineers

A contrast of 0.15 rubric points, about a quarter of the between-engineer standard deviation and the smallest difference we would regard as practically important, is detected with 85% power at 200 engineers and 98% at 300. A contrast of 0.10 needs about 300 engineers for 73%; at 200 the study has a coin-toss chance of detecting it. A contrast of 0.05 is beyond any cohort we can recruit. We will recruit 200 and say in advance that a null result rules out effects of 0.15 or more, not effects in general.

Rater reliability helps less than we expected, because the two-rater average already halves rater variance and diff-to-diff variation dominates. We will still train raters to a target of 0.8, but we will not spend recruitment budget on a third rater.

Result 2: how many companies

Engineer fixed effects absorb company differences in comprehension level, so company count barely matters when the effect is the same everywhere: power is 44% with six companies and 51 to 54% with ten or more. What matters is whether the effect itself varies by company, as it plausibly will with different tools, review cultures and codebases. With τ = 0.10, six companies give 24% power and 14 give 41%, because the clustered standard error now carries the between-company variation and the t reference has few degrees of freedom. We set a minimum of 14 companies, no company contributing more than 15% of engineers.

Result 3: engineers choose when to use the agent

The within-engineer design removes stable differences between people. It does not remove differences between diffs. If engineers reach for an agent more often on harder changes, and harder changes are harder to explain a month later, the contrast attributes difficulty to the agent. We simulated selection on complexity, adding λ·x_it to the logit of agent use.

With moderate selection (λ = 0.5) the estimate is −0.27, nearly three times the truth; with strong selection (λ = 1), −0.39. Adjusting for a complexity proxy as good as we think we can build from diff size, files touched and cyclomatic change (correlation 0.7 with true complexity) brings those to −0.19 and −0.26: about half the bias remains, as measurement-error theory predicts for an imperfect control. Covariate adjustment alone is not an adequate defence.

We therefore change the sampling, not just the model. Each month, instead of assessing one randomly chosen merged diff, we will assess a matched pair: one agent-authored and one self-authored diff from the same engineer, matched on the complexity proxy within a caliper. The comparison is then made within pairs of similar difficulty, and the proxy is used for matching and adjustment both. Where an engineer has no self-authored diff of comparable complexity that month, the month is recorded as unmatched and excluded from the primary analysis, which costs power and is reported.

Result 4: who leaves

Over a year some engineers will change jobs or stop responding. If dropout is unrelated to the effect, the fixed-effects estimate is unbiased for the cohort. If the engineers most harmed by agent use are more likely to leave, perhaps because they struggle, the completers understate the effect.

The bias is small at plausible dropout, and a tenth of the effect at 25% informative dropout. Because each engineer contributes observations until they leave, the fixed-effects model uses partial histories rather than discarding them, which is most of the protection. We will record the reason for every departure and pre-register a sensitivity analysis that re-weights completers by the inverse of their estimated probability of remaining.

Result 5: randomise engineers, not companies

The secondary question is whether one agent-free day per week for twelve weeks raises unassisted throughput. Randomising whole companies is operationally easier, since one manager agrees, and it avoids contamination between colleagues. We compared it with randomising 30% of engineers within each company, analysing both by regression of post-period log throughput on treatment and pre-period throughput, with company-clustered standard errors. The simulation includes a company-specific shock in the intervention months (a release crunch, a reorganisation) that a company-level design cannot separate from treatment.

Company randomisation treats four companies. Its clustered standard error, with four treated clusters, is too small, and the test rejects a true null 10% of the time at a nominal 5%; this is the few-treated-clusters problem documented by MacKinnon and Webb. Even with that inflation it has 40% power for a 15% effect. Randomising engineers within companies holds 3.5% false positives and has 86% power, because each company’s shock hits treated and control engineers alike. We accept the contamination risk, measure it by asking control engineers about their own agent use each week, and randomise within company.

What we are registering

• Sample: 200 engineers, at least 14 companies, no company above 15%.
• Primary outcome and model: equation (1), engineer fixed effects, company-clustered errors with a t_G−1 reference, on complexity-matched pairs.
• Smallest effect of interest: 0.15 rubric points; a null result is reported as ruling out effects at least that large, not as evidence of no effect.
• Raters: two per assessment, blind to authorship, trained to single-rater reliability of at least 0.8, reliability reported.
• Attrition: reasons recorded; inverse-probability-of-remaining sensitivity analysis pre-specified.
• Intervention: 30% of engineers randomised within company, twelve weeks, ANCOVA on log unassisted throughput, wild cluster bootstrap as a robustness check.
• Throughput: recorded and reported alongside comprehension, so that neither can be read without the other.

Method

Limitations

A design simulation is only as good as the threats it imagines. We assumed the effect is constant over the year, while the interesting possibility is that it grows; a study powered for a constant effect is somewhat over-powered for a growing one and blind to its shape unless the analysis models time. We assumed the agent indicator is measured without error, while in practice agent-authored is a matter of degree. We modelled selection on one complexity dimension; engineers may also choose agents for unfamiliar code, which is a different confounder our matching does not address. And the rubric itself has not yet been validated outside our own raters.

Simulation only. No participant data exists. The simulation script, its seed and the results file are under Artefacts, and the registration will cite this report.

Related work

The productivity evidence the study builds on is Peng and colleagues [1] and the field experiments of Cui and colleagues [2]; Bastani and colleagues [3] found the pattern we are looking for, better assisted performance and worse unassisted learning, in a school setting. Simulation-based power analysis follows Arnold and colleagues [4] and Green and MacLeod [5]. Fixed-effects estimation and its limits are standard [6]; clustered inference follows Cameron and Miller [7], and the few-treated-clusters problem that rules out company randomisation is MacKinnon and Webb [8]. Cluster-randomised design is Hayes and Moulton [9]. Missing-data mechanisms are Little and Rubin [10]; rater reliability, Shrout and Fleiss [11]; the case for registering all of this before data exist, Nosek and colleagues [12]. Our instruments are AWL-2023-01 [13] and AWL-2024-01 [14].

Artefacts

Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result.

References

1. Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590.
2. Cui, Z., Demirer, M., Jaffe, S., Musolff, L., Peng, S., & Salz, T. (2024). The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers. SSRN Working Paper.
3. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2024). Generative AI Can Harm Learning. SSRN Working Paper.
4. Arnold, B. F., Hogan, D. R., Colford, J. M., & Hubbard, A. E. (2011). Simulation Methods to Estimate Design Power: An Overview for Applied Research. BMC Medical Research Methodology, 11, 94.
5. Green, P., & MacLeod, C. J. (2016). SIMR: An R Package for Power Analysis of Generalized Linear Mixed Models by Simulation. Methods in Ecology and Evolution, 7(4).
6. Wooldridge, J. M. (2010). Econometric Analysis of Cross Section and Panel Data (2nd ed.). MIT Press.
7. Cameron, A. C., & Miller, D. L. (2015). A Practitioner’s Guide to Cluster-Robust Inference. Journal of Human Resources, 50(2).
8. MacKinnon, J. G., & Webb, M. D. (2018). The Wild Bootstrap for Few (Treated) Clusters. The Econometrics Journal, 21(2).
9. Hayes, R. J., & Moulton, L. H. (2017). Cluster Randomised Trials (2nd ed.). CRC Press.
10. Little, R. J. A., & Rubin, D. B. (2019). Statistical Analysis with Missing Data (3rd ed.). Wiley.
11. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass Correlations: Uses in Assessing Rater Reliability. Psychological Bulletin, 86(2).
12. Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The Preregistration Revolution. Proceedings of the National Academy of Sciences, 115(11).
13. Bhardwaj, M. (2023). An item bank for code comprehension: calibration, adaptive testing and fairness checks with item response theory. AI Work Lab Research, AWL-2023-01.
14. Bhardwaj, M. (2024). Reading the review state: a hidden Markov model of how engineers accept AI suggestions. AI Work Lab Research, AWL-2024-01.