Escaping the Echo Trap: On Credit Assignment Failure in Multi-turn LLM Self-Reflection

Despite the potential of multi-turn selfreflection to improve LLM reasoning, its effectiveness in practice is severely constrained by a failure mode we term the Echo Trap. Specifically, this phenomenon gives rise to two coupled problems: (1) the model becomes limited by its inherent capabilities and tends to repeat earlier reflections to preserve reward signals; (2) once such "copy" behavior is reinforced, the model ceases to try new strategies, leading to exploration collapse. We hypothesize that this issue is largely driven by imprecise credit assignment during training, as standard GRPO assigns rewards at the trajectory level, making it difficult to distinguish which reflection steps contribute to improved outcomes. To address this limitation, we propose a treestructured extension of GRPO for multi-turn self-reflection, which enables more accurate advantage estimation than standard trajectorylevel GRPO. Through extensive experiments, we analyze the Echo Trap and demonstrate that our method effectively mitigates behavior collapse and improves performance across multiple benchmarks. Our code is available at https://github.com/Flowersea37/TRAE