At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization

Pre-trained transformers have demonstrated remarkable generalization abilities, at times extending beyond the scope of their training data. Yet, real-world deployments often face unexpected or adversarial data that diverges from training data distributions. Without explicit mechanisms for handling such shifts, model reliability and safety degrade, urging more disciplined study of out-ofdistribution (OOD) settings for transformers. By systematic experiments, we present a mechanistic framework for delineating the precise contours of transformer model robustness. We find that OOD inputs, including subtle typos and jailbreak prompts, drive language models to operate on an increased number of spurious concepts in their internals. We leverage this device to quantify and understand the degree of distributional shift in prompts, enabling a mechanistically grounded fine-tuning strategy to increase the robustness of LLMs. Expanding the very notion of OOD from input data to a models private computational processesa new transformer diagnostic at inference timeis a critical step toward making AI systems safe for deployment across science, business, and government.