Benchmarking World-Model Learning with Environment-Level Queries

World models are central to building AI agents capable of flexible reasoning and planning. Yet current evaluations (i) test only properties measurable from observed interactions, such as nextframe prediction or task return, and (ii) do not test whether a learned model supports diverse queries about the environment. In contrast, humans build general-purpose models that can answer many different questions about an environment-including questions that require understanding global structure and counterfactual consequences. We propose WorldTest: a protocol for evaluating whether agents learn models that support multiple environment-level queries-questions whose answers depend on properties of the full environment, not just observed trajectories. Individually, these queries can target properties (e.g., reachability or the effects of interventions) that no single rollout distribution determines. Collectively, they assess model generality across query types. We instantiate WorldTest as AutumnBench, a benchmark of 43 interactive grid-world environments and 129 tasks across three query families for both humans and learning agents. Experiments with 517 human participants and five frontier models show that humans substantially outperform these models, a gap we attribute to differences in exploration and belief updating. AutumnBench provides a framework for evaluating world-model learning in grid-world environments with environmentlevel queries, and WorldTest provides a template for extending such evaluations to richer domains. Benchmarking World-Model Learning with Environment-Level Queries and reports the earliest timestep at which observations diverge from those predicted under the original environment; and (3) planning where the agent produces an action sequence to reach a target configuration, requiring reasoning about long-term consequences of actions. These three task types capture the capabilities illustrated in the kitchen example above: predicting what's in the covered pot, noticing when drawers are reorganized, and planning recipe steps. Together, these task-types yield 129 tasks across all environments covering core world-modeling capabilities including prediction, counterfactual reasoning, and planning. While not exhaustive, we designed AutumnBench to be extensible: new environments and challenges can introduce alternative dynamics (e.g., non-standard physics) or evaluate additional skills such as tool use or analogical reasoning. Finally, we validate AutumnBench through an empirical study involving 517 humans and five state-of-the-art AI models, demonstrating that the benchmark reliably exposes gaps between AI and human world-model learning. Contributions We make the following contributions: • We propose WorldTest, the first theoretical framework for testing world-model learning via environment-level queries through an interaction-based evaluation protocol. • We release AutumnBench, a concrete instantiation of WorldTest in a grid-world setting that operationalizes environment-level queries as automated behavioral tests. Additionally, AutumnBench follows the desiderata for novel games outlined by Ying et al. ( 2025 ) and is designed for easy extensibility. • We evaluate human and AI performance on Autumn-Bench, under the WorldTest framework, demonstrating that AutumnBench reliably exposes gaps between human reasoning and current state-of-the-art models.