We publish the measurements, including the ones we did not want.
Thirteen reports since 2023 on what agentic coding does to engineering capability, how to measure it without lying, and where a single beat of friction changes what an engineer retains. Every paper ships with its data, its instrument, and a limits section written before the marketing copy.
Everything else.
Explain-to-merge: where one beat of friction changes what sticks
Friction that teaches and friction that annoys are separated by about ninety seconds and one design decision about timing. We ran four placements of…
Read the paper →
Lines accepted predicts nothing: six productivity proxies against six-month capability
Every AI coding dashboard reports acceptance rate. We correlated six commonly-reported proxies against measured engineer capability six months later,…
Read the paper →
A concept tagger that does not lie: calibration across 41 repositories
The skill graph is only as honest as the classifier that populates it. We measured accuracy and calibration for four tagger designs across 41…
Read the paper →
Spaced retrieval for code: FSRS against 18,000 review events
Spaced repetition was built for vocabulary, where an item is atomic and a lapse is cheap. Code concepts are neither. We fitted FSRS to 18,240 code…
Read the paper →
Model-agnostic by construction: teaching quality across nine models
Bring-your-own-model is a permanent stance, not a launch limitation, which obliges us to show that the teaching layer works on models we do not…
Read the paper →
Bus factor as a leading indicator: 14 organisations, nine months
Concentration of understanding is easy to measure and easy to ignore until someone resigns. We tracked module-level comprehension concentration…
Read the paper →
What an org dashboard should refuse to show: aggregation limits on individual skill data
We built a skill graph accurate enough to be dangerous. This paper sets out the re-identification results that determined which views we ship, which…
Read the paper →
Designing a twelve-month cohort study of agentic coding: power, confounding and attrition by simulation
Next month we start following working engineers for a year to measure what agentic tools do to their understanding of the code they ship. Before…
Read the paper →
One run is an anecdote: variance in repository-level agent evaluations
Coding agents are now compared by a single resolve rate on a few hundred real GitHub issues, from one run. We simulated what repeated runs of the…
Read the paper →
Reading the review state: a hidden Markov model of how engineers accept AI suggestions
Acceptance rate tells you how often an engineer said yes to a suggestion, not whether they read it. We built a three-state hidden Markov model over…
Read the paper →
Pass@k is not a ranking: the statistics of code-model leaderboards
We needed to choose which models could sit underneath our teaching layer, and the evidence on offer was a leaderboard of single numbers on 164…
Read the paper →
An item bank for code comprehension: calibration, adaptive testing and fairness checks with item response theory
Before we can say anything about what AI assistance does to an engineer’s understanding, we need an instrument that measures understanding on a…
Read the paper →What we work on, and what we have not solved.
Each area lists the open problems honestly. If a question is unsolved we would rather say so here than have you discover it in the product.
Skill measurement
Whether an engineer can still explain, reproduce and modify the code they shipped, measured on their own repositories rather than on synthetic exercises.
Skill measurement
Open problems we are working on
- A comprehension instrument validated outside our own organisation
- Separating capability loss from delayed familiarity over windows longer than 90 days
- Measuring understanding of a system, not just of a diff
- Proxies that survive becoming a target for longer than three weeks
Selected work
- The comprehension gap: 200 engineers, 12 months, and what agentic coding did to recall · AWL-2026-06
- Lines accepted predicts nothing: six productivity proxies against six-month capability · AWL-2026-04
- An item bank for code comprehension: calibration, adaptive testing and fairness checks with item response theory · AWL-2023-01
- Reading the review state: a hidden Markov model of how engineers accept AI suggestions · AWL-2024-01
Learning science
Where to place friction so that it teaches, and how to schedule retrieval for concepts that behave nothing like vocabulary items.
Learning science
Open problems we are working on
- Retention for architectural concepts, currently 26 points behind syntactic ones
- Scheduling that fires on some merges rather than all of them without losing the effect
- Transfer: does retrieving a concept in one system strengthen it in another
- Long-interval behaviour beyond the horizon our event log can observe
Selected work
Agent evaluation
Making the teaching layer work on models we do not control, and making sure the classifier that populates the skill graph knows when it is guessing.
Agent evaluation
Open problems we are working on
- Calibration under genuine distribution shift, not just held-out repositories
- Concept adjacency errors, which are invisible because they look plausible
- Tagging code written through house-specific abstractions
- An evaluation that stays valid as models change every quarter
Selected work
- A concept tagger that does not lie: calibration across 41 repositories · AWL-2026-03
- Model-agnostic by construction: teaching quality across nine models · AWL-2026-01
- Pass@k is not a ranking: the statistics of code-model leaderboards · AWL-2023-02
- One run is an anecdote: variance in repository-level agent evaluations · AWL-2024-02
Org signals
Team-level consequences of concentrated understanding, and whether a warning arrives early enough that somebody can still act on it.
Org signals
Open problems we are working on
- Raising precision above 34% without losing the lead time that makes warnings useful
- Why 61% of warnings produce no action at all
- Distinguishing concentration caused by agents from concentration caused by staffing
- Signals for teams too small to aggregate safely
Selected work
Measurement integrity
What a product holding per-engineer comprehension data must refuse to show, and the re-identification work that decides where the line sits.
Measurement integrity
Open problems we are working on
- A formal privacy guarantee for the coverage views, which we do not yet claim
- Composition analysis across views that leak differently
- An answer better than suppression for organisations built out of pairs
- Modelling a patient adversary who accumulates monthly views for a year
Selected work
Study design
Deciding, before any data exists, how big a study must be, which observations to collect and which comparisons it can honestly support.
Study design
Open problems we are working on
- Matching assessed diffs on complexity without a validated complexity measure
- Designs that separate the effect of agent use from the choice to use it
- Analysis plans that stay valid when the tools change mid-study
Selected work
Who does the work.
Five research teams and three external advisors, thirteen papers. Every paper names the team that wrote it and an address that reaches them. Read more about the team →
Pre-registered, published, and allowed to be wrong.
Three commitments that constrain what we are able to claim.
Pre-registration
Primary outcomes and analysis plans are registered before data collection. It stops us discovering, after the fact, that the result we found was the one we meant to look for.
Publication regardless
No internal review can suppress a result for being commercially inconvenient. Three of our thirteen papers contain findings that are bad for us, including one that says our best metric collapses within a month of becoming a target.
Corrections in public
Papers carry a version and a changelog. When we get a number wrong we correct it in place and say what changed, rather than quietly replacing the file.