ACL Findings2025
FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inference
Runheng Liu, Xingchen Xiao, Heyan Huang, Zewen Chi, Zhijing Wu
Abstract
Retrieval-Augmented Language Modeling (RALM) by integrating large language models (LLM) with relevant documents from an external corpus is a proven methodology for enabling the LLM to generate information beyond the scope of its pre-training corpus. Previous work by retrieving a set of tokens iteratively with retrieved content prepending to the input poses a high run-time issue, which degrades the inference efficiency of the LLMs because they fail to use the Key-Value (KV) cache efficiently. We propose FLASHBACK, a modular RALM designed to improve the inference efficiency of RALM with the appending context pattern while maintaining decent performance after fine-tuning by Low-Rank Adaption. FLASHBACK appends retrieved documents at the end of the context to efficiently utilize the KV cache. We also introduce the Marking Token as two special prompt tokens for marking the appending context during fine-tuning. Our experiments show that FLASHBACK can improve language modeling performance in the perplexity metric. We proved that the Marking Token is a usable add-on when fine-tuning models on specific context patterns. By bypassing unnecessary recomputation, FLASHBACK achieves fast inference speed with long context input. The inference speed is up to 4× faster than the prepending counterpart on a 7B LLM (Llama 2) in the runtime test.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation cites LoRA: Low-Rank Adaptation of Large Language Models
- UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation cites Dense Passage Retrieval for Open-Domain Question Answering
- Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In cites Dense Passage Retrieval for Open-Domain Question Answering
Builds on6
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
- Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
- UPRISE: Universal Prompt Retrieval for Improving Zero-Shot EvaluationDaixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan et al.EMNLP 2023 · 28 citations
- Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-InZichun Yu, Chenyan Xiong, Shi Yu, Zhiyuan LiuACL 2023 · 15 citations
Related papers
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative DecodingGuanzheng Chen, Qilong Feng, Jinjie Ni, Xin Li et al.ICML 2025
- RA-DIT: Retrieval-Augmented Dual Instruction TuningXi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi et al.ICLR 2024 · 229 citations
- RetroLM: Retrieval-Augmented KVs for Long-Context ProcessingKun Luo, Zheng Liu, Shitao Xiao, Jiabei Chen et al.AAAI 2026
- Retrieval meets Long Context Large Language ModelsPeng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee et al.ICLR 2024 · 131 citations
- RefreshKV: Updating Small KV Cache During Long-form GenerationFangyuan Xu, Tanya Goyal, Eunsol ChoiACL 2025 · 6 citations
