Lune

ACL Findings2023

HuaSLIM: Human Attention Motivated Shortcut Learning Identification and Mitigation for Large Language models

Yuqi Ren, Deyi Xiong

Abstract

Large language models have made remarkable progress on a variety of NLP tasks. However, it has been found that they tend to rely on shortcut features that spuriously correlate with labels for prediction, which weakens their generalization on out-of-distribution samples. In this paper, we propose a human attention guided approach to identifying and mitigating shortcut learning, which encourages the LLM-based target model to learn relevant features. We define an attention-based measurement to capture both model and data bias and identify shortcut tokens by exploring both human and neural attention. In a self-distillation framework, we mitigate shortcut learning by dynamically adjusting the distillation temperature according to the detected shortcut tokens and estimated shortcut degree. Additionally, we utilize human attention as a supervisory signal to constrain large language models to pay more attention to relevant tokens. Experimental results on multiple NLP tasks show that our proposed method can effectively identify shortcut tokens, and significantly improve the robustness of large language models on OOD samples, while not undermining the performance on IID data.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Just Train Twice: Improving Group Robustness without Training Group Information cites An Investigation of Why Overparameterization Exacerbates Spurious Correlations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to ArtifactsHongyu Chen, Seraphina Goldfarb-TarrantACL 2025

Builds on17

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Just Train Twice: Improving Group Robustness without Training Group InformationEvan Zheran Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan et al.ICML 2021 · 683 citations
  3. The Pitfalls of Simplicity Bias in Neural NetworksHarshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain et al.NeurIPS 2020 · 503 citations
  4. An Investigation of Why Overparameterization Exacerbates Spurious CorrelationsShiori Sagawa, Aditi Raghunathan, Pang Wei Koh, Percy LiangICML 2020 · 436 citations
  5. Curriculum Temperature for Knowledge DistillationZheng Li, Xiang Li, Lingfeng Yang, Borui Zhao et al.AAAI 2023 · 277 citations

Related papers

  1. Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context LearningRuixiang Tang, Dehan Kong, Longtao Huang, Hui XueACL Findings 2023 · 31 citations
  2. LLM-Oriented Token-Adaptive Knowledge DistillationXurong Xie, Zhucun Xue, Jiafu Wu, Jian Li et al.AAAI 2026
  3. CAT: Causal Attention Tuning For Injecting Fine-grained Causal Knowledge into Large Language ModelsKairong Han, Wenshuo Zhao, Ziyu Zhao, Ye Jun Jian et al.EMNLP 2025 · 3 citations
  4. Efficient Unsupervised Shortcut Learning Detection and Mitigation in TransformersLukas Kuhn, Sari Sadiya, Jörg Schlötterer, Florian Buettner et al.ICCV 2025 · 1 citation
  5. Navigating the Shortcut Maze: A Comprehensive Analysis of Shortcut Learning in Text Classification by Language ModelsYuqing Zhou, Ruixiang Tang, Ziyu Yao, Ziwei ZhuEMNLP Findings 2024 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines