Lune

ACL Findings2023

Adaptation Approaches for Nearest Neighbor Language Models

Rishabh Bhardwaj, George Polovets, Monica Sunkara

Abstract

Semi-parametric Nearest Neighbor Language Models (kNN-LMs) have produced impressive gains over purely parametric LMs, by leveraging large-scale neighborhood retrieval over external memory datastores. However, there has been little investigation into adapting such models for new domains. This work attempts to fill that gap and suggests the following approaches for adapting kNN-LMs -1) adapting the underlying LM (using Adapters), 2) expanding neighborhood retrieval over an additional adaptation datastore, and 3) adapting the weights (scores) of retrieved neighbors using a learned Rescorer module. We study each adaptation strategy separately, as well as the combined performance improvement through ablation experiments and an extensive set of evaluations run over seven adaptation domains. Our combined adaptation approach consistently outperforms purely parametric adaptation and zero-shot (kNN-LM) baselines that construct datastores from the adaptation data. On average, we see perplexity improvements of 17.1% and 16% for these respective baselines, across domains.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Improving Language Models by Retrieving from Trillions of Tokens cites Generalization through Memorization: Nearest Neighbor Language Models
  • How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances cites LoRA: Low-Rank Adaptation of Large Language Models
  • How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances cites Generalization through Memorization: Nearest Neighbor Language Models
  • Nearest Neighbor Machine Translation cites Generalization through Memorization: Nearest Neighbor Language Models
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites LoRA: Low-Rank Adaptation of Large Language Models
  • Nearest Neighbor Speculative Decoding for LLM Generation and Attribution cites Improving Language Models by Retrieving from Trillions of Tokens
  • Nearest Neighbor Speculative Decoding for LLM Generation and Attribution cites Generalization through Memorization: Nearest Neighbor Language Models
  • Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory cites Improving Language Models by Retrieving from Trillions of Tokens
  • Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory cites Generalization through Memorization: Nearest Neighbor Language Models
  • Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memory cites Nearest Neighbor Machine Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Nearest Neighbor Speculative Decoding for LLM Generation and AttributionMinghan Li, Xilun Chen, Ari Holtzman, Beidi Chen et al.NeurIPS 2024 · 29 citations
  2. How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent AdvancesZihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad et al.EMNLP 2023 · 22 citations
  3. Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction MemoryGuangyue Peng, Tao Ge, Wen Luo, Wei Li et al.ACL 2025

Builds on8

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Improving Language Models by Retrieving from Trillions of TokensSebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai et al.ICML 2022 · 1,629 citations
  3. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
  4. Generalization through Memorization: Nearest Neighbor Language ModelsUrvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer et al.ICLR 2020 · 1,038 citations
  5. Nearest Neighbor Machine TranslationUrvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer et al.ICLR 2021 · 323 citations

Related papers

  1. Nearest Neighbor Zero-Shot InferenceWeijia Shi, Julian Michael, Suchin Gururangan, Luke ZettlemoyerEMNLP 2022 · 26 citations
  2. You can't pick your neighbors, or can you? When and How to Rely on Retrieval in the kNN-LMAndrew Drozdov, Shufan Wang, Razieh Rahimi, Andrew McCallum et al.EMNLP Findings 2022 · 10 citations
  3. Why do Nearest Neighbor Language Models Work?Frank F. Xu, Uri Alon, Graham NeubigICML 2023 · 33 citations
  4. Efficient Nearest Neighbor Language ModelsJunxian He, Graham Neubig, Taylor Berg-KirkpatrickEMNLP 2021
  5. Simple and Scalable Nearest Neighbor Machine TranslationYuhan Dai, Zhirui Zhang, Qiuzhi Liu, Qu Cui et al.ICLR 2023 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines