Hidden Breakthroughs in Language Model Training

Loss curves are smooth during most of model training, so visible discontinuities stand out as possible conceptual breakthroughs. These breakthroughs enable a deeper understanding of the model's concept structure, but only when they are properly identified. This paper argues that similar breakthroughs occur frequently throughout training, but they are obscured by a loss metric that collapses all variation into a single scalar. To find these hidden transitions, we introduce POLCA, a method for decomposing changes in loss along arbitrary bases of the low-rank training subspace. We use our method to identify clusters of samples that share similar changes in loss during training, disaggregating the overall loss into that of smaller groups of conceptually similar data. We validate our method on synthetic arithmetic and English language modeling, showing that POLCA recovers clusters that represent interpretable breakthroughs in the model's capabilities. We demonstrate the promise of these hidden breakthroughs as a tool for unsupervised interpretability.