The Limits of Predicting Agents from Behaviour

As the complexity of AI systems and their interactions with the world increases, generating explanations for their behaviour is important for safely deploying AI. For agents, the most natural abstractions for predicting behaviour attribute beliefs, intentions and goals to the system. If an agent behaves as if it has a certain goal or belief, then we can make reasonable predictions about how it will behave in novel situations, including those where comprehensive safety evaluations are untenable. How well can we infer an agent's beliefs from their behaviour, and how reliably can these inferred beliefs predict the agent's behaviour in novel situations? We provide a precise answer to this question under the assumption that the agent's behaviour is guided by a world model. Our contribution is the derivation of novel bounds on the agent's behaviour in new (unseen) deployment environments, which represent a theoretical limit for predicting intentional agents from behavioural data alone. We discuss the implications of these results for several research areas including fairness and safety. 1 Recent research suggests that an AI's behaviour, to the extent that it is consistent with rationality axioms, can be formally described by a (causal) world model (Halpern and Piermont, 2024) . The same conclusion can also be obtained for AIs capable of solving tasks in multiple environments (Richens and Everitt, 2024) . For large language models, there is increasing empirical evidence for the "world model" hypothesis, see e.g.