Lune

ACL Findings2025

Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement

Xiaofeng Zhou, Heyan Huang, Lizi Liao

Abstract

Large Language Models (LLMs) continue to set new standards in knowledge-intensive and complex reasoning tasks, yet their high computational demands limit widespread adoption. While distilling large models into smaller ones offers a sustainable solution, current techniques-such as static knowledge distillation, resource-intensive reinforcement learning from human feedback, or limited selfreflection-struggle to yield substantial and lasting performance gains. In this paper, we present a novel Debate and Reflect (D&R) framework that orchestrates multi-turn debates between smaller models and stronger teacher models, eliciting actionable feedback (e.g., error analysis, corrective strategies) to guide student models. Further, we introduce Treestructured Direct Preference Optimization (T-DPO) to efficiently leverage these debate logs, organizing interactions into a hierarchical format for effective training. Empirical evaluations across diverse NLP benchmarks demonstrate that our approach significantly improves smaller-model accuracy, robustness, and generalization, outperforming conventional baselines by a large margin.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Self-Refine: Iterative Refinement with Self-Feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on28

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
  4. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model SocietyGuohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin et al.NeurIPS 2023 · 1,975 citations
  5. Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations

Related papers

  1. daDPO: Distribution-Aware DPO for Distilling Conversational AbilitiesZhengze Zhang, Shiqi Wang, Yiqun Shen, Simin Guo et al.ACL Findings 2025 · 1 citation
  2. Democratizing Reasoning Ability: Tailored Learning from Large Language ModelZhaoyang Wang, Shaohan Huang, Yuxuan Liu, Jiahai Wang et al.EMNLP 2023 · 8 citations
  3. Teaching Small Language Models to Reason for Knowledge-Intensive Multi-Hop Question AnsweringXiang Li, Shizhu He, Fangyu Lei, Jun Yang et al.ACL Findings 2024 · 7 citations
  4. On the Generalization vs Fidelity Paradox in Knowledge DistillationSuhas Kamasetty Ramesh, Ayan Sengupta, Tanmoy ChakrabortyACL Findings 2025 · 1 citation
  5. Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge DistillationMinsang Kim, Seung Jun BaekICLR 2026 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines