Enhancing Human Experience in Human-Agent Collaboration: A Human-Centered Modeling Approach Based on Positive Human Gain

Existing game AI research mainly focuses on enhancing agents' abilities to win games, but this does not inherently make humans have a better experience when collaborating with these agents. For example, agents may dominate the collaboration and exhibit unintended or detrimental behaviors, leading to poor experiences for their human partners. In other words, most game AI agents are modeled in a "self-centered" manner. In this paper, we propose a "human-centered" modeling scheme for collaborative agents that aims to enhance the experience of humans. Specifically, we model the experience of humans as the goals they expect to achieve during the task. We expect that agents should learn to enhance the extent to which humans achieve these goals while maintaining agents' original abilities (e.g., winning games). To achieve this, we propose the Reinforcement Learning from Human Gain (RLHG) approach. The RLHG approach introduces a "baseline", which corresponds to the extent to which humans primitively achieve their goals, and encourages agents to learn behaviors that can effectively enhance humans in achieving their goals better. We evaluate the RLHG agent in the popular Multi-player Online Battle Arena (MOBA) game, Honor of Kings, by conducting real-world human-agent tests. Both objective performance and subjective preference results show that the RLHG agent provides participants better gaming experience. INTRODUCTION Recently, Reinforcement Learning (RL) has been widely used in developing Artificial Intelligence (AI) systems for games, developing various agents that perform at a human-level performance, such as AlphaGo (Silver et al., 2016; 2017 ) in Go, AlphaStar (Vinyals et al., 2019) in StarCraftII, OpenAI Five (OpenAI et al., 2019) in Dota2, and Wukong AI (Ye et al., 2020a) in Honor of Kings. To further expand the applications of these agents, researchers are exploring ways to improve their generalization to human partners (