Risk-Averse Total-Reward Reinforcement Learning

Risk-averse total-reward Markov Decision Processes (MDPs) offer a promising framework for modeling and solving undiscounted infinite-horizon objectives. Existing model-based algorithms for risk measures like the entropic risk measure (ERM) and entropic value-at-risk (EVaR) are effective in small problems, but require full access to transition probabilities. We propose a Q-learning algorithm to compute the optimal stationary policy for total-reward ERM and EVaR objectives with strong convergence and performance guarantees. The algorithm and its optimality are made possible by ERM's dynamic consistency and elicitability. Our numerical results on tabular domains demonstrate quick and reliable convergence of the proposed Q-learning algorithm to the optimal risk-averse value function.

ties in multi-stage optimization formulations [17][18][19]44]. In particular, dynamic decision-making with ERM allows for the existence of dynamic programming equations and Markov or stationary optimal policies. In addition, in this work, we leverage the fact that ERM is elicitable, which means that it can be estimated by solving a linear regression problem [7,11].

The second risk measure we consider is entropic value-at-risk (EVaR), which is defined for a given risk level α ∈ (0, 1) and x ∈ X as

and is extended to EVaR 0 [x] = ess inf[x] and EVaR 1 [x] = E[x] [2]. It is important to note that the supremum in (2) may not be attained even when x is a finite discrete random variable [3]. EVaR addresses several important shortcomings of ERM [17-19, 44]. In particular, EVaR is coherent and closely approximates popular and interpretable quantile-based risk measures, like VaR and CVaR [2, 19].

We formulate the decision process as a Markov Decision Process (MDP)