Modeling Unseen Environments with Language-guided Composable Causal Components in Reinforcement Learning

Generalization in reinforcement learning (RL) remains a significant challenge, especially when agents encounter novel environments with unseen dynamics. Drawing inspiration from human compositional reasoning-where known components are reconfigured to handle new situations-we introduce World Modeling with Compositional Causal Components (WM3C). This novel framework enhances RL generalization by learning and leveraging compositional causal components. Unlike previous approaches focusing on invariant representation learning or metalearning, WM3C identifies and utilizes causal dynamics among composable elements, facilitating robust adaptation to new tasks. Our approach integrates language as a compositional modality to decompose the latent space into meaningful components and provides theoretical guarantees for their unique identification under mild assumptions. Our practical implementation uses a masked autoencoder with mutual information constraints and adaptive sparsity regularization to capture high-level semantic information and effectively disentangle transition dynamics. Experiments on numerical simulations and real-world robotic manipulation tasks demonstrate that WM3C significantly outperforms existing methods in identifying latent processes, improving policy learning, and generalizing to unseen tasks. 1 Proposition 1 (Language-Controlled Components). Under the assumption that the graphical representation of the environment model is Markov and faithful (Spirtes et al., 2001; Glymour et al., 2019; Pearl, 2009) to the data, c i,t 2 is a minimal subset of state dimensions that are directly controlled by the language component l i and s j,t ∈ c i,t if and only if s j,t ̸ ⊥ ⊥ l i | a t-1:t , s t-1 , and s j,t ⊥ ⊥ l k k̸ =i | l i , a t-1:t , s t-1 .