Lune

ACL Findings2024

Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks

Verna Dankers, Ivan Titov

Abstract

Memorisation is a natural part of learning from real-world data: neural models pick up on atypical input-output combinations and store those training examples in their parameter space. That this happens is well-known, but how and where are questions that remain largely unanswered. Given a multi-layered neural model, where does memorisation occur in the millions of parameters? Related work reports conflicting findings: a dominant hypothesis based on image classification is that lower layers learn generalisable features and that deeper layers specialise and memorise. Work from NLP suggests this does not apply to language models, but has been mainly focused on memorisation of facts. We expand the scope of the localisation question to 12 natural language classification tasks and apply 4 memorisation localisation techniques. Our results indicate that memorisation is a gradual process rather than a localised one, establish that memorisation is task-dependent, and give nuance to the generalisation first, memorisation second hypothesis.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics cites Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
  • Quantifying Memorization Advantage in Code LLMs cites What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
  • Quantifying Memorization Advantage in Code LLMs cites Extracting Training Data from Large Language Models
  • Extracting alignment data in open models cites Extracting Training Data from Large Language Models
  • Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling cites Extracting Training Data from Large Language Models
  • Detecting Pretraining Data from Large Language Models cites Extracting Training Data from Large Language Models
  • Extracting Training Data from Large Language Models cites What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Extracting alignment data in open modelsFederico Barbero, Xiangming Gu, Christopher A. Choquette Choo, Chawin Sitawarin et al.ICML 2026 · 9 citations
  2. Arithmetic Without Algorithms: Language Models Solve Math with a Bag of HeuristicsYaniv Nikankin, Anja Reusch, Aaron Mueller, Yonatan BelinkovICLR 2025
  3. Quantifying Memorization Advantage in Code LLMsAlberick Euraste Djire, Abdoul Kader Kaboré, Jordan Samhi, Earl Barr et al.ICSE 2026
  4. ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution CiphersZhouxiang Fang, Aayush Mishra, Muhan Gao, Anqi Liu et al.EMNLP 2025

Builds on27

  1. Extracting Training Data from Large Language ModelsNicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski et al.USENIX Security 2021 · 2,866 citations
  2. Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
  3. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence EstimationVitaly Feldman, Chiyuan ZhangNeurIPS 2020 · 674 citations
  4. Detecting Pretraining Data from Large Language ModelsWeijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang et al.ICLR 2024 · 365 citations
  5. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language ModelsPeter Hase, Mohit Bansal, Been Kim, Asma GhandehariounNeurIPS 2023 · 307 citations

Related papers

  1. An Empirical Study of Memorization in NLPXiaosen Zheng, Jing JiangACL 2022 · 30 citations
  2. Can Neural Network Memorization Be Localized?Pratyush Maini, Michael Curtis Mozer, Hanie Sedghi, Zachary Chase Lipton et al.ICML 2023 · 82 citations
  3. Let's Agree to Agree: Neural Networks Share Classification Order on Real DatasetsGuy Hacohen, Leshem Choshen, Daphna WeinshallICML 2020 · 63 citations
  4. Exploring Memorization in Fine-tuned Language ModelsShenglai Zeng, Yaxin Li, Jie Ren, Yiding Liu et al.ACL 2024
  5. Identity Crisis: Memorization and Generalization Under Extreme OverparameterizationChiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer et al.ICLR 2020 · 96 citations
Dusk over the sea between two cliffs drawn in fine vertical lines