Fresh in memory: Training-order recency is linearly encoded in language model activations

We show that language models' activations linearly encode when information was learned during training. Our setup involves creating a model with a known training order by sequentially fine-tuning Llama-3.2-1B on six disjoint but otherwise similar datasets about named entities. We find that the average activations of test samples corresponding to the six training datasets encode the training order: when projected into a 2D subspace, these centroids are arranged exactly in the order of training and lie on a straight line. Further, we show that linear probes can accurately (∼90%) distinguish "early" vs. "late" entities, generalizing to entities unseen during the probes' own training. The model can also be fine-tuned to explicitly report an unseen entity's training stage (∼80% accuracy). Interestingly, the training-order encoding does not seem attributable to simple differences in activation magnitudes, losses, or model confidence. Our paper demonstrates that models are capable of differentiating information by its acquisition time, and carries significant implications for how they might manage conflicting data and respond to knowledge modifications. INTRODUCTION Average centroid difference (stage 1 -stage 6) Top PC orthogonal to x-axis Activation centroids (averages) for the six test datasets, across four independent training runs Synth -who Synth -stand for Natural -name Natural -meaning Actual training order D1 D2 D3 D4 D5 D6