We do not sell tokens and we do not take a markup, which means we have no revenue reason to prefer one model over another. It also means we owe users an honest answer to a question they should be asking: if I point this at a small local model, am I getting a worse education?
The short answer is that you get slightly worse questions and almost the same learning. This paper is the long answer.
Models and protocol
Nine models, three of them running locally on consumer hardware, spanning roughly a 40x range in cost per million tokens. Each model generated comprehension checks for the same 2,860 merges, and every check was graded by the same rubric-based grader held constant across arms.
Holding the grader constant while varying the generator is the design choice that makes this readable. If both sides move, you cannot tell whether a model wrote a better question or simply marked its own homework more generously.
Retention barely moves
The spread across all nine models is 7 percentage points, from 65% to 72%. For comparison, the placement effect in AWL-2026-05 was 27 points. The design of the teaching loop matters roughly four times more than which model runs inside it.
The weakest local model retained 65% against 71% for the strongest frontier model, which is 92% of the outcome at zero marginal cost on hardware the engineer already owns. We consider this the most important number in the paper, and it is the reason bring-your-own-model is a stance rather than a stopgap.
Question quality moves more than retention does
Three engineers blind-rated 900 generated questions for specificity, correctness and usefulness. Ratings ranged from 3.1 to 4.4 of 5, a much wider spread than retention. Frontier models asked noticeably sharper questions, particularly about a specific branch or edge case rather than the diff in general.
That sharpness did not convert proportionally into retention, which is consistent with AWL-2026-05: within a good placement, prompt content had a 4 point effect. The act of reconstructing your reasoning appears to do most of the work, largely independent of how elegantly you were asked.
| Model class | Retention (30d) | Question quality | Median latency | Refusal rate |
|---|---|---|---|---|
| Frontier hosted, largest | 71% | 4.4 | 3.1 s | 0.2% |
| Frontier hosted, mid | 72% | 4.2 | 2.4 s | 0.3% |
| Mid-tier hosted | 70% | 3.9 | 1.8 s | 0.9% |
| Small hosted | 68% | 3.5 | 1.1 s | 1.8% |
| Local 14B (M4 Max) | 69% | 3.6 | 6.2 s | 0.4% |
| Local 9B (M3 Pro) | 67% | 3.3 | 9.7 s | 0.6% |
| Local 7B (RTX 4070) | 65% | 3.1 | 7.4 s | 1.1% |
Small models were measurably worse at refusing to ask a question when the diff did not contain a taggable concept. The 7B model generated a question for 94% of diffs against 71% for the frontier model, and the surplus questions were mostly about formatting. We now gate generation behind the calibrated tagger from AWL-2026-03, which removes most of this gap.
What this costs the user
Because the model is the user's, the interesting cost is not ours. At March 2026 prices, running merge-time checks on every qualifying merge for a ten-engineer team costs roughly the figures below. The local column is the electricity, which we have not tried to price.
Limits
- Nine models at one point in time. This table will be wrong within two quarters and we will re-run it rather than defend it.
- We grouped results by model class rather than naming vendors per row. That is a deliberate choice to avoid a benchmark race, and it does reduce what a reader can verify.
- The grader is itself a model. A grader that shares a family with a generator may favour it, and we did not fully rule this out.
- Local latency figures are single-machine and sensitive to thermal state.
- Retention was measured over 30 days on 2,860 checks per arm, which gives us confidence intervals of roughly plus or minus 2 points. A 7 point spread is only a few intervals wide.
Per-model results, the blind rating instructions, generated question samples and the refusal analysis are published. Hardware thermal logs are available on request.
Related work
Holding the grader constant while varying the generator is the control Zheng and colleagues [1] show to be necessary once a model is used as a judge, and their self-preference finding is the specific bias we tried to design around. The local-model arms rely on quantisation behaviour documented by Dettmers and Zettlemoyer [2] and on the llama.cpp runtime [3]. Liang and colleagues [4] make the case for holistic, multi-metric evaluation rather than a single leaderboard number, which is why we report question quality, latency and refusal alongside retention. Our finding that loop design dominates model choice is consistent with the placement effect in AWL-2026-05 [5]. The refusal gate uses the calibrated tagger from AWL-2026-03 [6].
Artefacts
Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result.
References
- Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks.
- Dettmers, T., & Zettlemoyer, L. (2023). The Case for 4-bit Precision: k-bit Inference Scaling Laws. ICML.
- Gerganov, G., et al. (2023 onward). llama.cpp. Software.
- Liang, P., et al. (2023). Holistic Evaluation of Language Models. TMLR.
- Bhardwaj, M. (2026). Explain-to-merge: where one beat of friction changes what sticks. AI Work Lab Research, AWL-2026-05.
- Bhardwaj, M. (2026). A concept tagger that does not lie: calibration across 41 repositories. AI Work Lab Research, AWL-2026-03.
Bhardwaj, M. (2026). Model-agnostic by construction: teaching quality across nine models. AI Work Lab Research, AWL-2026-01.
