Compact Proofs of Model Performance via Mechanistic Interpretability
Jason Gross, Rajashree Agrawal, Thomas Kwa, Euan Ong, Chun Hei Yip, Alex Gibson, Soufiane Noubir, Lawrence Chan
Abstract
We propose using mechanistic interpretability -- techniques for reverse engineering model weights into human-interpretable algorithms -- to derive and compactly prove formal guarantees on model performance. We prototype this approach by formally proving accuracy lower bounds for a small transformer trained on Max-of-K, validating proof transferability across 151 random seeds and four values of K. We create 102 different computer-assisted proof strategies and assess their length and tightness of bound on each of our models. Using quantitative metrics, we find that shorter proofs seem to require and provide more mechanistic understanding. Moreover, we find that more faithful mechanistic understanding leads to tighter performance bounds. We confirm these connections by qualitatively examining a subset of our proofs. Finally, we identify compounding structureless errors as a key challenge for using mechanistic interpretability to generate compact proofs on model performance.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- A Toy Model of Universality: Reverse Engineering how Networks Learn Group Operations cites Language Models are Few-Shot Learners
- Towards a Unified and Verified Understanding of Group-Operation Networks cites A Toy Model of Universality: Reverse Engineering how Networks Learn Group Operations
- Transformers Learn In-Context by Gradient Descent cites Language Models are Few-Shot Learners
Cited by top-tier papers2
- Path Channels and Plan Extension Kernels: a Mechanistic Description of Planning in a Sokoban RNNMohammad Taufeeque, Aaron David Tucker, Adam Gleave, Adrià Garriga-AlonsoICLR 2026 · 5 citations
- Towards a Unified and Verified Understanding of Group-Operation NetworksWilson Wu, Louis Jaburi, Jacob Drori, Jason GrossICLR 2025
Builds on8
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- AI2: Safety and Robustness Certification of Neural Networks with Abstract InterpretationTimon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov et al.S&P 2018 · 987 citations
- Transformers Learn In-Context by Gradient DescentJohannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento et al.ICML 2023 · 729 citations
- A Toy Model of Universality: Reverse Engineering how Networks Learn Group OperationsBilal Chughtai, Lawrence Chan, Neel NandaICML 2023 · 144 citations
- Einops: Clear and Reliable Tensor Manipulations with Einstein-like NotationAlex RogozhnikovICLR 2022 · 124 citations
Related papers
- Validating Mechanistic Interpretations: An Axiomatic ApproachNils Palumbo, Ravi Mangal, Zifan Wang, Saranya Vijayakumar et al.ICML 2025
- Learning Transformer ProgramsDan Friedman, Alexander Wettig, Danqi ChenNeurIPS 2023 · 59 citations
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 SmallKevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris et al.ICLR 2023 · 50 citations
- Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable GuaranteesItamar Hadad, Guy Katz, Shahaf BassanICLR 2026 · 10 citations
- Towards Automated Circuit Discovery for Mechanistic InterpretabilityArthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim et al.NeurIPS 2023 · 861 citations
