ACL Findings2026
Breaking Contextual Inertia: Reinforcement Learning with Single-Turn Anchors for Stable Multi-Turn Interaction
Xingwu Chen, Zhanqiu Zhang, Steven Y. Guo, Difan Zou
Abstract
While LLMs demonstrate strong reasoning capabilities when provided with full information in a single turn, they exhibit substantial vulnerability in multi-turn interactions. Specifically, when information is revealed incrementally or requires updates, models frequently fail to integrate new constraints, leading to a collapse in performance compared to their single-turn baselines. We term the root cause as Contextual Inertia: a phenomenon where models rigidly adhere to previous reasoning traces. Even when users explicitly provide corrections or new data in later turns, the model ignores them, preferring to maintain consistency with its previous (incorrect) reasoning path. To address this, we introduce Reinforcement Learning with Single-Turn Anchors (RLSTA), a generalizable training approach designed to stabilize multi-turn interaction across diverse scenarios and domains. RLSTA leverages the model's superior singleturn capabilities as stable internal anchors to provide reward signals. By aligning multi-turn responses with these anchors, RLSTA empowers models to break contextual inertia and selfcalibrate their reasoning based on the latest information. Experiments show that RLSTA significantly outperforms standard fine-tuning and abstention-based methods. Notably, our method exhibits strong cross-domain generalization (e.g., math to code) and proves effective even without external verifiers, highlighting its potential for general-domain applications.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- LLMs Get Lost In Multi-Turn Conversation cites MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
- ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds on14
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- LLMs Get Lost In Multi-Turn ConversationPhilippe Laban, Hiroaki Hayashi, Yingbo Zhou, Jennifer NevilleICLR 2026 · 491 citations
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language FeedbackXingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen et al.ICLR 2024 · 308 citations
- ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLYifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine et al.ICML 2024 · 163 citations
- Enhancing Chat Language Models by Scaling High-quality Instructional ConversationsNing Ding, Yulin Chen, Bokai Xu, Yujia Qin et al.EMNLP 2023 · 95 citations
Related papers
- Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention RewardsMing Li, Pei Chen, Zhenhao Zhang, Tao Yang et al.ACL 2026 · 3 citations
- Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to DeliberationZhiwei Zhang, Xiaomin Li, Yudi Lin, Hui Liu et al.ICLR 2026 · 13 citations
- The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use AgentsWeihao Xuan, Qingcheng Zeng, Heli Qi, Yunze Xiao et al.ACL 2026 · 4 citations
- SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated ReasoningZhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li et al.ICLR 2026 · 152 citations
- In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-FeedbackMingye Zhu, Yi Liu, Zheren Fu, Quan Wang et al.AAAI 2026 · 1 citation
