Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory
Guangyue Peng, Tao Ge, Wen Luo, Wei Li, Houfeng Wang
Abstract
Semiparametric language models (LMs) have shown promise in various Natural Language Processing (NLP) tasks. However, they utilize non-parametric memory as static storage, which lacks learning capability and remains disconnected from the internal information flow of the parametric models, limiting scalability and efficiency. Based on recent interpretability theories of LMs, we reconceptualize the non-parametric memory represented by k NN-LM as a learnable Mixture-of-Neighbors Induction Memory (MoNIM), which synergizes the induction capabilities of attention heads with the memorization strength of feed-forward networks (FFN). By integrating into the model’s information flow, MoNIM functions as an FFN-like bypass layer within the Transformer architecture, enabling effective learning of new knowledge. Extensive experiments demonstrate that MoNIM is a retentive and scalable continual learner in both data-and model-wise, enhancing the scalability and continual learning performance of semiparametric LMs. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Improving Language Models by Retrieving from Trillions of Tokens cites Generalization through Memorization: Nearest Neighbor Language Models
- Improving Language Models by Retrieving from Trillions of Tokens cites Retrieval Augmented Language Model Pre-Training
- Nearest Neighbor Machine Translation cites Generalization through Memorization: Nearest Neighbor Language Models
- Retrieval Augmented Language Model Pre-Training cites Generalization through Memorization: Nearest Neighbor Language Models
Cited by top-tier papers1
Builds on23
- Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
- Improving Language Models by Retrieving from Trillions of TokensSebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai et al.ICML 2022 · 1,629 citations
- Generalization through Memorization: Nearest Neighbor Language ModelsUrvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer et al.ICLR 2020 · 1,038 citations
- S2ORC: The Semantic Scholar Open Research CorpusKyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney et al.ACL 2020 · 424 citations
- Nearest Neighbor Machine TranslationUrvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer et al.ICLR 2021 · 323 citations
Related papers
- Adaptation Approaches for Nearest Neighbor Language ModelsRishabh Bhardwaj, George Polovets, Monica SunkaraACL Findings 2023 · 7 citations
- Integrated Episodic and Semantic Memory via Modulating Transformer FeedForward LayersYiqun Yao, Xiang Li, Xin Jiang, Xuezhi Fang et al.ICML 2026
- Decouple knowledge from paramters for plug-and-play language modelingXin Cheng, Yankai Lin, Xiuying Chen, Dongyan Zhao et al.ACL Findings 2023 · 4 citations
- Self-Updatable Large Language Models by Integrating Context into Model ParametersYu Wang, Xinshuang Liu, Xiusi Chen, Sean O'Brien et al.ICLR 2025
- Scalable Language Model with Generalized Continual LearningBohao Peng, Zhuotao Tian, Shu Liu, Ming-Chang Yang et al.ICLR 2024 · 36 citations
