Every product in this category eventually discovers that it can improve learning outcomes by making the tool worse to use. The interesting question is not whether friction works. It is how little of it you can get away with, and precisely when to spend it.
We had a specific version of this problem. The comprehension check in our product asks the engineer to explain a generated diff in their own words before it merges. That is a real cost imposed on someone trying to close a ticket. This study is our attempt to justify the cost honestly, and to find the cheapest placement that still earns it.
Design
Ninety-four engineers across nine teams, randomised into five arms. Every arm saw an identical prompt generated by the same model with the same rubric. The only variable was when it appeared.
Randomising within team rather than between teams cost us statistical power and bought us protection against the thing most likely to ruin a study like this, which is that one team has a strong review culture and another does not.
Merge time wins, and not by a little
Merge-time placement produced 68% retention at 30 days against 41% for the no-prompt control. Post-generation placement reached 64%, statistically indistinguishable from merge time on retention alone. Pre-generation placement was barely better than control at 45%, and T+24h landed at 52%.
Pre-generation failing is the result we should have predicted and did not. Asking someone to predict what code will do before they have seen it sounds like a generation effect, which is well supported in the literature. In practice engineers treated it as a formality standing between them and the thing they wanted, and answered it in the shortest way that would dismiss the dialogue.
The placements that worked are the ones where the engineer had already read the code and was being asked to reconstruct it without looking. That is retrieval practice. The placements that failed asked for either prediction with no referent or recall so late that the context was gone.
What it costs, and who quits
Retention is only half a decision. A check that works perfectly and gets switched off in week three has an effect size of zero in production, and this is where the two leading placements separated sharply.
Merge-time placement added a median of 94 seconds per merge and was voluntarily abandoned by 6% of participants. Post-generation placement added only 71 seconds but was abandoned by 31%, a fivefold difference. The interviews were unanimous about why: interrupting generation breaks the flow of building, while a check at merge time lands at a boundary where the engineer has already mentally shifted into review.
| Placement | 30-day retention | Added time (median) | 3-week abandonment | Review rework |
|---|---|---|---|---|
| No prompt (control) | 41% | 0s | n/a | baseline |
| Pre-generation | 45% | 62s | 18% | +2% |
| Post-generation | 64% | 71s | 31% | -4% |
| Merge time | 68% | 94s | 6% | -11% |
| T+24 hours | 52% | 88s | 24% | -1% |
The review rework column was not a primary outcome and is the finding we would most like to see replicated. Merge-time checks were associated with an 11% reduction in PRs needing a follow-up fix inside two weeks. If that holds up, a meaningful part of the 94 seconds pays for itself immediately rather than in some future quarter when the engineer is more capable.
Prompt content mattered much less than we expected
We varied prompt phrasing across three styles within the merge-time arm: open explanation, targeted question about a specific branch, and a request to predict the effect of a proposed modification. The spread between them was 4 percentage points of retention, well inside the noise of the placement effect.
- Open explanation: 67% retention, most disliked in interviews, highest variance in answer quality.
- Targeted branch question: 71% retention, best received, hardest to generate reliably.
- Modification prediction: 66% retention, strongest correlation with the 90-day measure.
- Placement effect across arms: 27 percentage points. Content effect within an arm: 4.
This is convenient for engineering and slightly deflating for the research programme. The thing we spent most of our model budget on, generating a good question, matters less than the thing that costs nothing, choosing when to ask it.
Limits
- Eleven weeks is short for a retention claim. The 30-day window is nested inside a study barely twice its length.
- Voluntary abandonment in a study where participants know they are observed almost certainly understates real-world abandonment.
- Nine teams, all at companies already paying for a coding agent. Teams that have not adopted agents may respond differently to being interrupted by one.
- The review rework result is a secondary outcome found in exploratory analysis and should be treated as a hypothesis, not a finding.
- We did not test a placement that fires on some merges rather than all of them, which is what a shipping product should probably do.
Arm assignments, the prompt bank, blind scoring instructions and the abandonment event log are published. Merge diffs are not.
Related work
The placement result is a direct application of the testing effect [1] and of Bjork's desirable difficulty [2], with the important qualification that difficulty has to be placed where retrieval is possible. Our failed pre-generation arm is best read against the generation-effect literature [3], which predicts a benefit we did not observe, most likely because the participants had no referent to generate against. Kluger and DeNisi [4] anticipate the abandonment asymmetry we measured: feedback interventions that interrupt task execution are systematically rejected even when effective. The spacing follow-up is in AWL-2026-02 [5], and the comprehension instrument is documented in AWL-2026-06 [6].
Artefacts
Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result.
References
- Roediger, H. L., & Karpicke, J. D. (2006). The Power of Testing Memory. Perspectives on Psychological Science, 1(3).
- Bjork, R. A. (1994). Memory and Metamemory Considerations in the Training of Human Beings. In Metacognition. MIT Press.
- Slamecka, N. J., & Graf, P. (1978). The Generation Effect: Delineation of a Phenomenon. Journal of Experimental Psychology, 4(6).
- Kluger, A. N., & DeNisi, A. (1996). The Effects of Feedback Interventions on Performance. Psychological Bulletin, 119(2).
- Bhardwaj, M. (2026). Spaced retrieval for code: FSRS against 18,000 review events. AI Work Lab Research, AWL-2026-02.
- Bhardwaj, M. (2026). The comprehension gap: 200 engineers, 12 months, and what agentic coding did to recall. AI Work Lab Research, AWL-2026-06.
Bhardwaj, M. (2026). Explain-to-merge: where one beat of friction changes what sticks. AI Work Lab Research, AWL-2026-05.
