Lune

ACL Findings2025

All That Glitters is Not Gold: Improving Robust Retrieval-Augmented Language Models with Fact-Centric Preference Alignment

Jia Hao, Chunhong Zhang, Jiarun Liu, Haiyu Zhao, Zhiqiang Zhan, Zheng Hu

Abstract

Retrieval-augmented language model (RALM) relies on retrieved external knowledge to generate responses, resulting in vulnerability in the face of retrieval results with noisy documents. Previous works integrate additional filters or finetune Large Language Models (LLMs) to learn adaptive retrieval to reduce the performance damage of noisy documents. However, prior noise filtering may lead to the loss of crucial information, and these methods do not focus on distracting documents with high semantic relevance, which is the most challenging problem. In this study, we propose a training method for fact-centric preference alignment (FPA) to improve the ability of LLMs to directly extract useful information from noisy retrieval results without prior filtering. Our method performs positive document mining based on factual consistency and uses LLMs self-generated synthetic data as training data without manual annotation. We evaluate our FPA on four question answering benchmarks, and the experimental results demonstrate that our method achieves significant improvement with a small scale of training data. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Training language models to follow instructions with human feedback
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on24

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  4. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  5. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations

Related papers

  1. MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented GenerationChia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan et al.ACL 2025
  2. Making RALM Robust to Irrelevant Contexts via Layer Knowledge Guided AttentionWeijie Shi, Hao Chen, Jiaming Li, Yao Zhao et al.ACL Findings 2025 · 1 citation
  3. REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question AnsweringYuhao Wang, Ruiyang Ren, Junyi Li, Xin Zhao et al.EMNLP 2024 · 12 citations
  4. FAITH: Factuality Alignment through Integrating Trustworthiness and HonestnessXiaoning Dong, Chengyan Wu, Yajie Wen, Yu Chen et al.ACL Findings 2026
  5. ROPO: Robust Preference Optimization for Large Language ModelsXize Liang, Chao Chen, Shuang Qiu, Jie Wang et al.ICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines