Model and Reinforcement Learning for Markov Games with Risk Preferences
Wenjie Huang, Pham Viet Hai, William Benjamin Haskell
Abstract
We motivate and propose a new model for non-cooperative Markov game which considers the interactions of risk-aware players. This model characterizes the time-consistent dynamic “risk” from both stochastic state transitions (inherent to the game) and randomized mixed strategies (due to all other players). An appropriate risk-aware equilibrium concept is proposed and the existence of such equilibria is demonstrated in stationary strategies by an application of Kakutani's fixed point theorem. We further propose a simulation-based Q-learning type algorithm for risk-aware equilibrium computation. This algorithm works with a special form of minimax risk measures which can naturally be written as saddle-point stochastic optimization problems, and covers many widely investigated risk measures. Finally, the almost sure convergence of this simulation-based algorithm to an equilibrium is demonstrated under some mild conditions. Our numerical experiments on a two player queuing game validate the properties of our model and algorithm, and demonstrate their worth and applicability in real life competitive decision-making.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Related papers
- Risk-Averse Total-Reward Reinforcement LearningXihong Su, Jia Lin Hau, Gersi Doko, Kishan Panaganti et al.NeurIPS 2025
- Planning and Learning in Average Risk-aware MDPsWeikai Wang, Erick DelageNeurIPS 2025 · 3 citations
- Tractable Multi-Agent Reinforcement Learning through Behavioral EconomicsEric Mazumdar, Kishan Panaganti, Laixi ShiICLR 2025
- Provably Convergent Actor-Critic in Risk-averse MARLYizhou Zhang, Eric MazumdarICML 2026
- Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in RegretYingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang et al.NeurIPS 2020 · 87 citations
