Nearest Neighbor Machine Translation
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, Mike Lewis
Abstract
We introduce -nearest-neighbor machine translation (NN-MT), which predicts tokens with a nearest neighbor classifier over a large datastore of cached examples, using representations from a neural translation model for similarity search. This approach requires no additional training and scales to give the decoder direct access to billions of examples at test time, resulting in a highly expressive model that consistently improves performance across many settings. Simply adding nearest neighbor search improves a state-of-the-art German-English translation model by 1.5 BLEU. NN-MT allows a single model to be adapted to diverse domains by using a domain-specific datastore, improving results by an average of 9.2 BLEU over zero-shot transfer, and achieving new state-of-the-art results---without training on these domains. A massively multilingual model can also be specialized for particular language pairs, with improvements of 3 BLEU for translating from English into German and Chinese. Qualitatively, NN-MT is easily interpretable; it combines source and target context to retrieve highly relevant examples.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
79 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token cites Generalization through Memorization: Nearest Neighbor Language Models
- xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token cites Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory
- xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Retrieval-Augmented Diffusion Models cites Generalization through Memorization: Nearest Neighbor Language Models
- Retrieval-Augmented Diffusion Models cites Accelerating Large-Scale Inference with Anisotropic Vector Quantization
- End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering cites Generalization through Memorization: Nearest Neighbor Language Models
- End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- BASE Layers: Simplifying Training of Large, Sparse Models cites Generalization through Memorization: Nearest Neighbor Language Models
- Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory cites Generalization through Memorization: Nearest Neighbor Language Models
- Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks cites Generalization through Memorization: Nearest Neighbor Language Models
Cited by top-tier papers79
- BASE Layers: Simplifying Training of Large, Sparse ModelsMike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal et al.ICML 2021 · 382 citations
- Retrieval-Augmented Diffusion ModelsAndreas Blattmann, Robin Rombach, Kaan Oktay, Jonas Müller et al.NeurIPS 2022 · 239 citations
- End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question AnsweringDevendra Singh Sachan, Siva Reddy, William L. Hamilton, Chris Dyer et al.NeurIPS 2021 · 197 citations
- Lift Yourself Up: Retrieval-augmented Text Generation with Self-MemoryXin Cheng, Di Luo, Xiuying Chen, Lemao Liu et al.NeurIPS 2023 · 177 citations
- xRAG: Extreme Context Compression for Retrieval-augmented Generation with One TokenXin Cheng, Xun Wang, Xingxing Zhang, Tao Ge et al.NeurIPS 2024 · 156 citations
Builds on8
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Generalization through Memorization: Nearest Neighbor Language ModelsUrvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer et al.ICLR 2020 · 1,038 citations
- Accelerating Large-Scale Inference with Anisotropic Vector QuantizationRuiqi Guo, Philip Sun, Erik Lindgren, Quan Geng et al.ICML 2020 · 539 citations
- Improving Massively Multilingual Neural Machine Translation and Zero-Shot TranslationBiao Zhang, Philip Williams, Ivan Titov, Rico SennrichACL 2020 · 213 citations
- With Little Power Comes Great ResponsibilityDallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia et al.EMNLP 2020 · 76 citations
Related papers
- Subset Retrieval Nearest Neighbor Machine TranslationHiroyuki Deguchi, Taro Watanabe, Yusuke Matsui, Masao Utiyama et al.ACL 2023 · 7 citations
- Non-Parametric Unsupervised Domain Adaptation for Neural Machine TranslationXin Zheng, Zhirui Zhang, Shujian Huang, Boxing Chen et al.EMNLP Findings 2021 · 23 citations
- Simple and Scalable Nearest Neighbor Machine TranslationYuhan Dai, Zhirui Zhang, Qiuzhi Liu, Qu Cui et al.ICLR 2023 · 9 citations
- Learning Kernel-Smoothed Machine Translation with Retrieved ExamplesQingnan Jiang, Mingxuan Wang, Jun Cao, Shanbo Cheng et al.EMNLP 2021 · 1 citation
- Revisiting Source Context in Nearest Neighbor Machine TranslationXuanhong Li, Peng Li, Po HuEMNLP 2023
