Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learning

Offline reinforcement learning (RL) has progressed with return-conditioned supervised learning (RCSL), but its lack of stitching ability remains a limitation. We introduce Q-Aided Conditional Supervised Learning (QCS), which effectively combines the stability of RCSL with the stitching capability of Q-functions. By analyzing Q-function over-generalization, which impairs stable stitching, QCS adaptively integrates Q-aid into RCSL's loss function based on trajectory return. Empirical results show that QCS significantly outperforms RCSL and valuebased methods, consistently achieving or exceeding the maximum trajectory returns across diverse offline RL benchmarks. The project page is available at https://beanie00.com/publications/qcs.