Multi-Agent Reinforcement Learning with General Utilities via Decentralized Shadow Reward Actor-Critic

Reinforcement learning (RL) is a framework for directly estimating the parameters of a controller through repeated interaction with the environment, and has gained attention for its ability to alleviate the need for a physically exact model across a number of domains, such as robotic manipulation [12] , web services [31] , and various games [25] . In RL, an agent in a given state takes an action, and transits to another according to a Markov transition density, whereby a reward informing the merit of the action is revealed by the environment. Mathematically, this setting may be encapsulated by a Markov Decision Process (MDP) [23] , in which the one seeks to select the action sequence to maximize the long-term accumulation of rewards. In many domains, multiple agents interact in order to obtain favorable outcomes, as in finance [17] , social networks [9], and games [27] . In multi-agent RL (MARL) and more generally, stochastic games, a key question is the payoff structure [2] . We focus on common payoffs among agents, i.e., the utility of the team is the sum of local utilities [4], which contrasts with competitive settings where one agent's gain is another's loss, or combinations thereof [18] . Whereas typically cooperative MARL defines the global utility as the average over agents' local reward accumulations, here we define a new mechanism for cooperation that permits agents to incorporate risk-sensitivity [3, 22] , prior experience [1], or exploration [8]. The usual common-payoff setting focuses on global cumulative return of rewards, which is a linear function of the the state-action occupancy measure. By contrast, the aforementioned decision-making goals define nonlinear functions of the state-action occupancy measure [10]. Such functions, we call general utilities, have recently yielded impressive performance in practice via prioritizing exploration [7], risk-sensitivity [21], and prior