M³HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality

Designing effective reward functions in multiagent reinforcement learning (MARL) is a significant challenge, often leading to suboptimal or misaligned behaviors in complex, coordinated environments. We introduce Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality (M 3 HF), a novel framework that integrates multi-phase human feedback of mixed quality into the MARL training process. By involving humans with diverse expertise levels to provide iterative guidance, M 3 HF leverages both expert and non-expert feedback to continuously refine agents' policies. During training, we strategically pause agent learning for human evaluation, parse feedback using large language models to assign it appropriately and update reward functions through predefined templates and adaptive weights by using weight decay and performance-based adjustments. Our approach enables the integration of nuanced human insights across various levels of quality, enhancing the interpretability and robustness of multiagent cooperation. Empirical results in challenging environments demonstrate that M 3 HF significantly outperforms state-of-the-art methods, effectively addressing the complexities of reward design in MARL and enabling broader human participation in the training process. Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality " The r ose chef shoul d get t he t omat o f i r st , t hen t he gr een chef can t ake i t and cut i t as qui ckl y as possi bl e. The bl ue chef can t ake t he pl at e and put t he cut s on i t . " Human Feedback Rol l out Vi deo Mul t i -agent RL ... l ambda obs, act : ( 1 i f act == 5 el se 0) + ( 1 i f obs[ 9] == obs[ 0] and obs[ 10] == obs[ 1] el se 0) l ambda obs, act : ( 1 i f act == 5 el se 0) + ( 1 i f obs[ 2] == 1 el se 0) Func t i on W ei ght s Adj us t ment l ambda obs, act : ( -sqr t ( ( obs[ 19] -obs[ 0] ) * * 2 + ( obs[ 20] -obs[ 1] ) * * 2) ) Rewar d Func t i on Pool s ...