EMNLP Findings2025
When Inverse Data Outperforms: Exploring the Pitfalls of Mixed Data in Multi-Stage Fine-Tuning
Mengyi Deng, Xin Li, Tingyu Zhu, Zhicheng Yang, Zhijiang Guo, Wei Wang
Abstract
Existing work has shown that o1-level performance can be achieved with limited data distillation, but most existing methods focus on unidirectional supervised fine-tuning (SFT), overlooking the intricate interplay between diverse reasoning patterns. In this paper, we construct r1k, a high-quality reverse reasoning dataset derived by inverting 1,000 forward examples from s1k (Muennighoff et al., 2025), and examine how SFT and Direct Preference Optimization (DPO) affect alignment under bidirectional reasoning objectives. SFT on r1k yields a 1.6%-6.8% accuracy improvement over s1k across evaluated benchmarks. However, naively mixing forward and reverse data during SFT weakens the directional distinction. Although DPO can partially recover this distinction, it also suppresses less preferred reasoning paths by shifting the probability mass toward irrelevant outputs. These findings suggest that mixed reasoning data introduce conflicting supervision signals, underscoring the need for robust and direction-aware alignment strategies. Our code and data are available at: https://github.com/16demi/ ReasonAlign-analysis.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Self-Refine: Iterative Refinement with Self-Feedback cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Let's Verify Step by Step cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection cites Self-Refine: Iterative Refinement with Self-Feedback
Builds on15
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Let's Verify Step by StepHunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards et al.ICLR 2024 · 3,045 citations
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-ReflectionAkari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil et al.ICLR 2024 · 1,798 citations
Related papers
- DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise OptimizationMengyi Deng, Zhiwei Li, Xin Li, Tingyu Zhu et al.ACL Findings 2026
- As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback LossXin Mao, Huimin Xu, Feng-Lin Li, Ziqi Jin et al.ICLR 2025
- Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM ReasoningShuyao Xu, Cheng Peng, Jiangxuan Long, Weidi Xu et al.ACL 2026 · 3 citations
- MetaGDPO: Alleviating Catastrophic Forgetting with Metacognitive Knowledge Through Group Direct Preference OptimizationLanxue Zhang, Yuqiang Xie, Fang Fang, Fanglong Dong et al.AAAI 2026 · 1 citation
- Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning ModelsHuifeng Yin, Yu Zhao, Minghao Wu, Xuanfan Ni et al.ACL 2025 · 14 citations
