A concept tagger that does not lie: calibration across 41 repositories

A skill graph that says an engineer understands a concept they do not understand is worse than no skill graph. It will schedule the wrong review, skip the check that would have caught the gap, and tell a manager something false with a number attached. Everything downstream depends on the tagger, so the tagger's failure modes are the product's failure modes.

This paper is about a trade we made deliberately and would make again: we shipped the less accurate classifier.

Corpus and labelling

We hand-labelled 12,400 code spans across 41 repositories, spanning Python, TypeScript, Go, Java, Rust, C++ and SQL. Labels came from a 214-concept taxonomy covering things an engineer can plausibly be said to understand or not: backpressure, transaction isolation, memory ownership, cache invalidation, and so on.

Splitting by repository rather than by span is the detail that makes this evaluation worth reading. A span-level split lets the model see other functions from the same file and inflates every number reported. We measured a 6.2 point accuracy drop moving from span-level to repository-level splits on the same model, which is roughly the size of the difference between our four candidate designs.

Four designs

• Design A: single large-model call per span, confidence from token logprobs.
• Design B: same model, structured chain of custody requiring the model to cite the specific lines supporting each tag.
• Design C: small fine-tuned classifier over an embedding of the span plus its enclosing scope.
• Design D: Design B, with an ensemble of three prompts and confidence taken from inter-prompt agreement rather than logprobs.

Design A is the most accurate at 92.1% and the least honest. Its expected calibration error is 0.19, and the shape of the error is the dangerous kind: when it is wrong it is usually confident. Design D reaches 88.4% with an ECE of 0.04. Its confidence scores mean something.

Why calibration beats accuracy here

Suppose tagger A is right 92% of the time and wrong with full confidence. Tagger D is right 88% of the time, but a score of 0.9 means it is right about nine times in ten and a score of 0.5 means it is guessing. Tagger D is far more useful, because a calibrated score is something the rest of the system can act on differentially.

Under this three-band policy the calibrated tagger produces 61% fewer incorrect graph updates than the accuracy-optimised one, because most of its errors fall into bands where the system does not act on them. The accurate-but-overconfident model has no such escape hatch: its errors arrive dressed as certainties and go straight into the graph.

Where it still fails

Errors concentrate in three places, and none of them are evenly distributed across your codebase, which means the tagger is more wrong about some engineers than others.

• Novel abstractions: a concept implemented through a house-specific wrapper is frequently missed. Accuracy on repositories with heavy internal frameworks drops to 81%.
• Concept adjacency: distinguishing optimistic concurrency from transaction isolation, or backpressure from rate limiting, accounts for 34% of all errors.
• Generated and vendored code: correctly tagged as containing a concept, wrongly attributed to the engineer who merged it. We now exclude paths matching vendor and generated patterns, which loses real signal.
• Very short spans: below eight lines, calibration degrades faster than accuracy.

The adjacency errors are the ones we consider most serious, because they are invisible. A wrongly-tagged concept in the same neighbourhood as the right one still produces a plausible-looking skill graph and a review schedule that is subtly aimed at the wrong target.

Limits

• The 214-concept taxonomy is ours. A different taxonomy would produce different numbers and we cannot say how different.
• 41 repositories skews toward web and backend services. No embedded, kernel, or numerical code.
• Post-adjudication labeller agreement of 0.89 sets a ceiling on measurable accuracy that we are within four points of.
• Calibration was measured on the same repository distribution as training. Calibration under genuine distribution shift is untested and is the obvious next study.
• Cost figures are at May 2026 API prices and will not age well.

The taxonomy, the labelling guide, per-language confusion matrices and the reliability diagrams are published. The labelled corpus itself is available to researchers under a data agreement, since it contains code from private repositories.

Related work

Our decision to prefer calibration over raw accuracy follows Guo and colleagues [1], who documented that modern neural networks are badly overconfident, and uses the reliability-diagram and ECE apparatus from Naeini and colleagues [2]. Brier [3] is the origin of the proper-scoring intuition underneath the whole argument. The repository-level split matters for the reasons Kapoor and Narayanan [4] set out in their survey of leakage in machine-learning-based science, and the specific hazard of span-level leakage in code models is documented by Allamanis [5]. The three-band decision policy is a selective-prediction design in the sense of Geifman and El-Yaniv [6]. Downstream consumers of the tagger are described in AWL-2026-02 [7].

Artefacts

Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result.

References

1. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. ICML.
2. Naeini, M. P., Cooper, G. F., & Hauskrecht, M. (2015). Obtaining Well Calibrated Probabilities Using Bayesian Binning. AAAI.
3. Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review, 78(1).
4. Kapoor, S., & Narayanan, A. (2023). Leakage and the Reproducibility Crisis in Machine-Learning-Based Science. Patterns, 4(9).
5. Allamanis, M. (2019). The Adverse Effects of Code Duplication in Machine Learning Models of Code. Onward!.
6. Geifman, Y., & El-Yaniv, R. (2017). Selective Classification for Deep Neural Networks. NeurIPS.
7. Bhardwaj, M. (2026). Spaced retrieval for code: FSRS against 18,000 review events. AI Work Lab Research, AWL-2026-02.