Lune

ICLR2025Top-tier venue

Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains

Vighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba, Shuang Li, Igor Mordatch

Abstract

Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. To improve models beyond the training data, recent works have explored how LLMs can be used to generate synthetic data for autonomous self-improvement. However, successive steps of self-improvement can reach a point of diminishing returns. In this work, we propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models. A group of language models, all starting from the same base model, are independently specialized by updating each one using data generated through multiagent interactions among the models. By training each model on independent sets of data, we illustrate how this approach enables specialization across models and diversification over the set of models. As a result, our overall system is able to preserve diverse reasoning chains and autonomously improve over many more rounds of fine-tuning than single-agent self-improvement methods. We quantitatively illustrate the efficacy of the approach across a wide suite of reasoning tasks. Project website at https://llm-multiagent-ft.github.io

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

28 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Language Models are Few-Shot Learners
  • Thought Communication in Multiagent Collaboration cites Improving Factuality and Reasoning in Language Models through Multiagent Debate
  • Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning cites Language Models are Few-Shot Learners
  • ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning cites Improving Factuality and Reasoning in Language Models through Multiagent Debate
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
  • Visual Instruction Tuning cites Language Models are Few-Shot Learners
  • SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
  • SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning cites Improving Factuality and Reasoning in Language Models through Multiagent Debate
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers28

  1. OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task AutomationMengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie et al.NeurIPS 2025 · 158 citations
  2. ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement LearningZiyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song et al.NeurIPS 2025 · 76 citations
  3. SiriuS: Self-improving Multi-agent Systems via Bootstrapped ReasoningWanjia Zhao, Mert Yüksekgönül, Shirley Wu, James Y. ZouNeurIPS 2025 · 49 citations
  4. Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language ModelsMickel Liu, Liwei Jiang, Yancheng Liang, Simon Du et al.ICML 2026 · 34 citations
  5. Thought Communication in Multiagent CollaborationYujia Zheng, Zhuokai Zhao, Zijian Li, Yaqi Xie et al.NeurIPS 2025 · 31 citations

Builds on25

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  4. Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
  5. Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations

Related papers

  1. Large Language Models Can Self-ImproveJiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu et al.EMNLP 2023 · 184 citations
  2. A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard CurriculaChenruo Liu, Yijun Dong, Yiqiu Shen, Qi LeiICML 2026
  3. The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding?Yutao Sun, Mingshuai Chen, Tiancheng Zhao, Ruochen Xu et al.ACL Findings 2025 · 1 citation
  4. I Learn Better If You Speak My Language: Understanding the Superior Performance of Fine-Tuning Large Language Models with LLM-Generated ResponsesXuan Ren, Biao Wu, Lingqiao LiuEMNLP 2024 · 1 citation
  5. Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMsHaritz Puerto, Tilek Chubakov, Xiaodan Zhu, Harish Tayyar Madabushi et al.ACL 2025 · 13 citations
Dusk over the sea between two cliffs drawn in fine vertical lines