Lune

ACL Findings2024

Improving Attributed Text Generation of Large Language Models via Preference Learning

Dongfang Li, Zetian Sun, Baotian Hu, Zhenyu Liu, Xinshuo Hu, Xuebo Liu, Min Zhang

Abstract

Large language models have been widely adopted in natural language processing, yet they face the challenge of generating unreliable content. Recent works aim to reduce misinformation and hallucinations by resorting to attribution as a means to provide evidence (i.e., citations). However, current attribution methods usually focus on the retrieval stage and automatic evaluation that neglect mirroring the citation mechanisms in human scholarly writing to bolster credibility. In this paper, we address these challenges by modeling the attribution task as preference learning and introducing an Automatic Preference Optimization (APO) framework. First, we create a curated collection for post-training with 6,330 examples by collecting and filtering from existing datasets. Second, considering the high cost of labeling preference data, we further propose an automatic method to synthesize attribution preference data resulting in 95,263 pairs. Moreover, inspired by the human citation process, we further propose a progressive preference optimization method by leveraging fine-grained information. Extensive experiments on three datasets (i.e., ASQA, StrategyQA, and ELI5) demonstrate that APO achieves state-of-the-art citation F1 with higher answer quality. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • Advancing Large Language Model Attribution through Self-Improving cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering cites Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • A³: Automatic Alignment Framework for Attributed Text Generation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • A³: Automatic Alignment Framework for Attributed Text Generation cites Training language models to follow instructions with human feedback
  • Let's Verify Step by Step cites Training language models to follow instructions with human feedback
  • Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling cites Let's Verify Step by Step
  • Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling cites Language Models are Few-Shot Learners
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward ModelingJunyi Li, Hwee Tou NgACL 2025 · 5 citations
  2. Advancing Large Language Model Attribution through Self-ImprovingLei Huang, Xiaocheng Feng, Weitao Ma, Liang Zhao et al.EMNLP 2024 · 2 citations
  3. AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context AttributionFengyuan Liu, Nikhil Kandpal, Colin RaffelICLR 2025
  4. MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question AnsweringSeokwon Song, Minsu Park, Gunhee KimAAAI 2026
  5. A³: Automatic Alignment Framework for Attributed Text GenerationYue Wang, Haoke Zhang, Juntao Li, Jinxiong Chang et al.ACL 2025

Builds on16

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  4. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  5. Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations

Related papers

  1. Learning Fine-Grained Grounded Citations for Attributed Large Language ModelsLei Huang, Xiaocheng Feng, Weitao Ma, Yuxuan Gu et al.ACL Findings 2024 · 10 citations
  2. CiteGuard: Faithful Citation Attribution for LLMs via Retrieval-Augmented ValidationYee Man Choi, Xuehang Guo, Yi R. Fung, Qingyun WangACL 2026 · 7 citations
  3. Enabling Large Language Models to Generate Text with CitationsTianyu Gao, Howard Yen, Jiatong Yu, Danqi ChenEMNLP 2023 · 152 citations
  4. AttributionBench: How Hard is Automatic Attribution Evaluation?Yifei Li, Xiang Yue, Zeyi Liao, Huan SunACL Findings 2024 · 4 citations
  5. Training Language Models to Generate Text with Citations via Fine-grained RewardsChengyu Huang, Zeqiu Wu, Yushi Hu, Wenya WangACL 2024 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines