Lune

NeurIPS2023Top-tier venue

ResMem: Learn what you can and memorize the rest

Zitong Yang, Michal Lukasik, Vaishnavh Nagarajan, Zonglin Li, Ankit Singh Rawat, Manzil Zaheer, Aditya Krishna Menon, Sanjiv Kumar

Abstract

The impressive generalization performance of modern neural networks is attributed in part to their ability to implicitly memorize complex training patterns. Inspired by this, we explore a novel mechanism to improve model generalization via explicit memorization. Specifically, we propose the residual-memorization (ResMem) algorithm, a new method that augments an existing prediction model (e.g., a neural network) by fitting the model's residuals with a k-nearest neighbor based regressor. The final prediction is then the sum of the original model and the fitted residual regressor. By construction, ResMem can explicitly memorize the training labels, even when the base model has low capacity. We start by formulating a stylized linear regression problem and rigorously show that ResMem results in a more favorable test risk over a base linear neural network. Then, we empirically show that ResMem consistently improves the test set generalization of the original prediction model across standard vision and natural language processing benchmarks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Improving Language Models by Retrieving from Trillions of Tokens cites Generalization through Memorization: Nearest Neighbor Language Models
  • Improving Language Models by Retrieving from Trillions of Tokens cites Accelerating Large-Scale Inference with Anisotropic Vector Quantization
  • The Pitfalls of Memorization: When Memorization Hurts Generalization cites What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
  • Synthetic Bootstrapped Pretraining cites Improving Language Models by Retrieving from Trillions of Tokens
  • Synthetic Bootstrapped Pretraining cites Generalization through Memorization: Nearest Neighbor Language Models
  • Synthetic Bootstrapped Pretraining cites Accelerating Large-Scale Inference with Anisotropic Vector Quantization
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. DeltaPhi: Physical States Residual Learning for Neural Operators in Data-Limited PDE SolvingXihang Yue, Yi Yang, Linchao ZhuNeurIPS 2025 · 5 citations
  2. Synthetic Bootstrapped PretrainingZitong Yang, Aonan Zhang, Hong Liu, Tatsunori Hashimoto et al.ICLR 2026 · 3 citations
  3. The Pitfalls of Memorization: When Memorization Hurts GeneralizationReza Bayat, Mohammad Pezeshki, Elvis Dohmatob, David Lopez-Paz et al.ICLR 2025 · 1 citation

Builds on10

  1. Improving Language Models by Retrieving from Trillions of TokensSebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai et al.ICML 2022 · 1,629 citations
  2. Test-Time Training with Self-Supervision for Generalization under Distribution ShiftsYu Sun, Xiaolong Wang, Zhuang Liu, John Miller et al.ICML 2020 · 1,220 citations
  3. Generalization through Memorization: Nearest Neighbor Language ModelsUrvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer et al.ICLR 2020 · 1,038 citations
  4. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence EstimationVitaly Feldman, Chiyuan ZhangNeurIPS 2020 · 674 citations
  5. Accelerating Large-Scale Inference with Anisotropic Vector QuantizationRuiqi Guo, Philip Sun, Erik Lindgren, Quan Geng et al.ICML 2020 · 539 citations

Related papers

  1. Meta-NeighborhoodsSiyuan Shan, Yang Li, Junier B. OlivaNeurIPS 2020 · 17 citations
  2. How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?Wei Huang, Andi Han, Yujin Song, Yilan Chen et al.NeurIPS 2025 · 4 citations
  3. Random Label Prediction Heads for Studying Memorization in Deep Neural NetworksMarlon Becker, Jonas Konrad, Luis Garcia Rodriguez, Benjamin RisseICLR 2026
  4. The Curious Case of Benign MemorizationSotiris Anagnostidis, Gregor Bachmann, Lorenzo Noci, Thomas HofmannICLR 2023 · 1 citation
  5. Generalizablity of Memorization Neural NetworkLijia Yu, Xiao-Shan Gao, Lijun Zhang, Yibo MiaoNeurIPS 2024 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines