Embracing Anisotropy: Turning Massive Activations into Interpretable Control Knobs for Large Language Models

Large Language Models (LLMs) exhibit highly anisotropic internal representations, often characterized by massive activations, a phenomenon where a small subset of feature dimensions possesses magnitudes significantly larger than the rest. While prior works view these extreme dimensions primarily as artifacts to be managed, we propose a distinct perspective: these dimensions serve as intrinsic interpretable functional units arising from domain specialization. Specifically, we propose a simple magnitude-based criterion to identify Domain-Critical Dimensions in a trainingfree manner. Our analyses reveal that such dimensions behave as interpretable semantic detectors for symbolic/quantitative patterns or domain-specific terms. In addition, we introduce Critical Dimension Steering, which applies activation steering exclusively to the identified dimensions. Empirical results show that this approach outperforms conventional wholedimension steering in domain adaptation and jailbreaking scenarios. 1 The following are multiple choice questions (with answers) about high school biology. You must respond with a single alphabet character. Energy is harvested during cellular respiration in stages. Which of the following correctly states which phase of cellular respiration harvests the most energy and the correct explanation why? A. The most energy is released during the Krebs cycle because it is here that pyruvate is completely ... B. ... both FADH2 and NADH are produced. Each of those molecules will release 2 ATPs ... C. ... phosphorylation because in addition to the phosphorylation of ADP into ATP ... D. ... during oxidative phosphorylation because H2O is completely broken down into H + and O2. Answer: 334 (Top 1) 1068 (Top 4) 2106 (Top 7) Act. Token Act. Token Act. Token 2048.0 <bos> 65.0 _the 43.8 H 324.0 The 64.0 _the 43.0 _ATP 147.0 _biology 62.0 _is 42.2 _ATP 139.0 Energy 61.0 _to 41.5 _respiration 138.0 _Krebs 59.5 _to 41.2 _energy 137.0 _cellular 58.8 _the 41.2 _ATP 132.0 _school 57.5 _the 41.2 _phosphorylation 130.0 _harvested 55.8 _that 40.8 _ADP 130.0 _cellular 55.0 _The 38.5 _NAD 130.0 _respiration 54.8 _in 38.2 _phosphorylation The following are multiple choice questions (with answers) about high school mathematics. You must respond with a single alphabet character. Suppose 5 different integers are randomly chosen from between 20 and 69, inclusive. What is the probability that they each have a different tens digit? A. 14 B. 13 C. 100052969 D. 250052969 Answer: The following are multiple choice questions (with answers) about high school mathematics. You must respond with a single alphabet character. An equilateral triangle has sides of 12 inches. What is the approximate area of the triangle?