EMNLP Findings2022
You can't pick your neighbors, or can you? When and How to Rely on Retrieval in the kNN-LM
Andrew Drozdov, Shufan Wang, Razieh Rahimi, Andrew McCallum, Hamed Zamani, Mohit Iyyer
Abstract
Retrieval-enhanced language models (LMs), which condition their predictions on text retrieved from large external datastores, have recently shown significant perplexity improvements compared to standard LMs. One such approach, the kNN-LM, interpolates any existing LM's predictions with the output of a knearest neighbors model and requires no additional training. In this paper, we explore the importance of lexical and semantic matching in the context of items retrieved by kNN-LM. We find two trends: (1) the presence of large overlapping n-grams between the datastore and evaluation set plays an important factor in strong performance, even when the datastore is derived from the training data; and (2) the kNN-LM is most beneficial when retrieved items have high semantic similarity with the query. Based on our analysis, we define a new formulation of the kNN-LM that uses retrieval quality to assign the interpolation coefficient. We empirically measure the effectiveness of our approach on two English language modeling datasets, Wikitext-103 and PG-19. Our re-formulation of the kNN-LM is beneficial in both cases, and leads to nearly 4% improvement in perplexity on the Wikitext-103 test set.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
12 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Improving Language Models by Retrieving from Trillions of Tokens cites Generalization through Memorization: Nearest Neighbor Language Models
- Improving Language Models by Retrieving from Trillions of Tokens cites Deduplicating Training Data Makes Language Models Better
- Improving Language Models by Retrieving from Trillions of Tokens cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances cites Generalization through Memorization: Nearest Neighbor Language Models
- How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore cites Improving Language Models by Retrieving from Trillions of Tokens
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore cites Generalization through Memorization: Nearest Neighbor Language Models
- SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore cites Deduplicating Training Data Makes Language Models Better
- Unlimiformer: Long-Range Transformers with Unlimited Length Input cites Improving Language Models by Retrieving from Trillions of Tokens
- Unlimiformer: Long-Range Transformers with Unlimited Length Input cites Generalization through Memorization: Nearest Neighbor Language Models
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks cites Generalization through Memorization: Nearest Neighbor Language Models
- Nearest Neighbor Speculative Decoding for LLM Generation and Attribution cites Improving Language Models by Retrieving from Trillions of Tokens
- Nearest Neighbor Speculative Decoding for LLM Generation and Attribution cites Generalization through Memorization: Nearest Neighbor Language Models
- Nearest Neighbor Speculative Decoding for LLM Generation and Attribution cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Cited by top-tier papers12
- Unlimiformer: Long-Range Transformers with Unlimited Length InputAmanda Bertsch, Uri Alon, Graham Neubig, Matthew GormleyNeurIPS 2023 · 176 citations
- SILO Language Models: Isolating Legal Risk In a Nonparametric DatastoreSewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi et al.ICLR 2024 · 91 citations
- Repoformer: Selective Retrieval for Repository-Level Code CompletionDi Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan et al.ICML 2024 · 78 citations
- Nearest Neighbor Speculative Decoding for LLM Generation and AttributionMinghan Li, Xilun Chen, Ari Holtzman, Beidi Chen et al.NeurIPS 2024 · 29 citations
- How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent AdvancesZihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad et al.EMNLP 2023 · 22 citations
Builds on19
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Improving Language Models by Retrieving from Trillions of TokensSebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai et al.ICML 2022 · 1,629 citations
- Generalization through Memorization: Nearest Neighbor Language ModelsUrvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer et al.ICLR 2020 · 1,038 citations
- Deduplicating Training Data Makes Language Models BetterKatherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang et al.ACL 2022 · 844 citations
- Compressive Transformers for Long-Range Sequence ModellingJack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier et al.ICLR 2020 · 833 citations
Related papers
- Why do Nearest Neighbor Language Models Work?Frank F. Xu, Uri Alon, Graham NeubigICML 2023 · 33 citations
- kNN-LM Does Not Improve Open-ended Text GenerationShufan Wang, Yixiao Song, Andrew Drozdov, Aparna Garimella et al.EMNLP 2023 · 3 citations
- Adaptation Approaches for Nearest Neighbor Language ModelsRishabh Bhardwaj, George Polovets, Monica SunkaraACL Findings 2023 · 7 citations
- Efficient Nearest Neighbor Language ModelsJunxian He, Graham Neubig, Taylor Berg-KirkpatrickEMNLP 2021
- Neuro-Symbolic Language Modeling with Automaton-augmented RetrievalUri Alon, Frank F. Xu, Junxian He, Sudipta Sengupta et al.ICML 2022 · 79 citations
