EMNLP Findings2023
Self-Knowledge Guided Retrieval Augmentation for Large Language Models
Abstract
Large language models (LLMs) have shown superior performance without task-specific finetuning. Despite the success, the knowledge stored in the parameters of LLMs could still be incomplete and difficult to update due to the computational costs. As complementary, retrieval-based methods can offer nonparametric world knowledge and improve the performance on tasks such as question answering. However, we find that the retrieved knowledge does not always help and even has a negative impact on original responses occasionally. To better make use of both internal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is also called "self-knowledge") and propose Self-Knowledge guided Retrieval augmentation (SKR), a simple yet effective method which can let LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions. We evaluate SKR on multiple datasets and demonstrate that it outperforms chain-of-thought based and fully retrieval-based methods by using either In-structGPT or ChatGPT. Code is released at https://github.com/THUNLP-MT/SKR.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
29 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Right this way: Can VLMs Guide Us to See More to Answer Questions? cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks cites Large Language Models are Zero-Shot Reasoners
- Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Training language models to follow instructions with human feedback
- Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering cites Repoformer: Selective Retrieval for Repository-Level Code Completion
- Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering cites Large Language Models are Zero-Shot Reasoners
- Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
Cited by top-tier papers29
- Repoformer: Selective Retrieval for Repository-Level Code CompletionDi Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan et al.ICML 2024 · 78 citations
- Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question AnsweringZhihua Wen, Zhiliang Tian, Zexin Jian, Zhen Huang et al.NeurIPS 2024 · 29 citations
- Parametric Retrieval Augmented GenerationWeihang Su, Yichen Tang, Qingyao Ai, Junxi Yan et al.SIGIR 2025 · 25 citations
- Right this way: Can VLMs Guide Us to See More to Answer Questions?Li Liu, Diji Yang, Sijia Zhong, Kalyana Suma Sree Tholeti et al.NeurIPS 2024 · 20 citations
- Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning TasksHuanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang et al.AAAI 2025 · 17 citations
Builds on33
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
Related papers
- SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented GenerationZijun Yao, Weijian Qi, Liangming Pan, Shulin Cao et al.ACL 2025
- Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMsOded Ovadia, Menachem Brief, Moshik Mishaeli, Oren ElishaEMNLP 2024 · 89 citations
- Fine-grained Knowledge Enhancement for Retrieval-Augmented GenerationJingxuan Han, Zhendong Mao, Yi Liu, Yexuan Che et al.ACL Findings 2025 · 3 citations
- KBM: Delineating Knowledge Boundary for Adaptive Retrieval in Large Language ModelsZhen Zhang, Xinyu Wang, Yong Jiang, Zile Qiao et al.EMNLP Findings 2025
- Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back HomeViktor Moskvoretskii, Maria Marina, Mikhail Salnikov, Nikolay Ivanov et al.ACL 2025 · 22 citations
