Explainable Reinforcement Learning through a Causal Lens

Prominent theories in cognitive science propose that humans understand and represent the knowledge of the world through causal relationships. In making sense of the world, we build causal models in our mind to encode cause-effect relations of events and use these to explain why new events happen by referring to counterfactuals -things that did not happen. In this paper, we use causal models to derive causal explanations of the behaviour of model-free reinforcement learning agents. We present an approach that learns a structural causal model during reinforcement learning and encodes causal relationships between variables of interest. This model is then used to generate explanations of behaviour based on counterfactual analysis of the causal model. We computationally evaluate the model in 6 domains and measure performance and task prediction accuracy. We report on a study with 120 participants who observe agents playing a real-time strategy game (Starcraft II) and then receive explanations of the agents' behaviour. We investigate: 1) participants' understanding gained by explanations through task prediction; 2) explanation satisfaction and 3) trust. Our results show that causal model explanations perform better on these measures compared to two other baseline explanation models. Driven by lack of trust from users and proposed regulations, there are many calls for Artificial Intelligence (AI) systems to become more transparent, interpretable and explainable. This has renewed the interest in Explainable AI (XAI), which has been explored since the expert systems era (Chandrasekaran, Tanner, and Josephson 1989) . A key pillar of XAI is explanation, a justification given for decisions and actions of the system. However, much research and practice in XAI pays little attention to people as intended users of these systems (Miller 2018b). If we are to build systems that are capable of providing 'good' explanations, it is plausible that explanation models should mimic models of human explanation (De Graaf and Malle 2017). Thus, to build