Lune

ACL Findings2025

Spectral Insights into Data-Oblivious Critical Layers in Large Language Models

Xuyuan Liu, Lei Hsiung, Yaoqing Yang, Yujun Yan

Abstract

Understanding how feature representations evolve across layers in large language models (LLMs) is key to improving their interpretability and robustness. While recent studies have identified critical layers linked to specific functions or behaviors, these efforts typically rely on data-dependent analyses of fine-tuned models, limiting their use to post-hoc settings. In contrast, we introduce a data-oblivious approach to identify intrinsic critical layers in pre-fine-tuned LLMs by analyzing representation dynamics via Centered Kernel Alignment (CKA). We show that layers with significant shifts in representation space are also those most affected during fine-tuning-a pattern that holds consistently across tasks for a given model. Our spectral analysis further reveals that these shifts are driven by changes in the top principal components, which encode semantic transitions from rationales to conclusions. We further apply these findings to two practical scenarios: efficient domain adaptation, where fine-tuning critical layers leads to greater loss reduction compared to non-critical layers; and backdoor defense, where freezing them reduces attack success rates by up to 40%.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! cites Finetuned Language Models are Zero-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Representational Similarity and Model Behavior in Multi-Agent InteractionYujin Potter, Seun Eisape, Shiyang Lai, Alexander Huth et al.ICML 2026
  2. Representation Interventions Enable Lifelong Knowledge Memory Control in LLMsXuyuan Liu, Shengyu Chen, Xinshuai Dong, Yanchi Liu et al.ACL 2026

Builds on21

  1. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  2. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
  3. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen et al.ICLR 2024 · 1,104 citations
  4. Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and DepthThao Nguyen, Maithra Raghu, Simon KornblithICLR 2021 · 323 citations
  5. The Internal State of an LLM Knows When It's LyingAmos Azaria, Tom M. MitchellEMNLP Findings 2023 · 177 citations

Related papers

  1. Critical-CoT: A Robust Defense Framework against Reasoning-Level Backdoor Attacks in Large Language ModelsVu Tuan Truong, Long Bao LeACL 2026
  2. OASIS: Mitigating Harmful Fine-tuning Attacks on LLMs via Orthogonal and Adaptive Safety Alignment StrategyJiayu Tang, Guowei Peng, Qiuhao Xie, Yuning Yang et al.ACL 2026
  3. Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank ModificationsBoyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie et al.ICML 2024 · 215 citations
  4. Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?Zi Liang, Haibo Hu, Qingqing Ye, Yaxin Xiao et al.ICML 2025
  5. Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMsZiqian Zhong, Aditi RaghunathanICLR 2026 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines