Mixture of Meta-Policies for Cross-Environment Meta-Reinforcement Learning

Meta-Reinforcement Learning (Meta-RL) aims to enable agents to rapidly adapt to new tasks by leveraging prior experience from related ones. However, existing approaches typically assume consistent state and action spaces across tasks, limiting the ability of Meta-RL methods to generalize to environments with diverse morphologies, dynamics, and reward structures, common in real-world applications. To address this, we propose Mixture of Meta-Policies (MoMP), a modular and scalable Meta-RL framework designed for effective knowledge transfer across structurally heterogeneous environments. MoMP represents policies as sparse combinations of shared submodules, where each submodule is itself an attention-based policy block. During meta-training, MoMP learns to specialize and reuse these attention modules by optimizing their task-conditioned activation across multiple environments. Once trained, the shared module can be plugged into new agents to accelerate learning in novel tasks. Sparse activation enables targeted reuse of prior knowledge while mitigating interference, improving both adaptation speed and long-term retention. Experiments across diverse MuJoCo agents show that MoMP significantly outperforms strong meta-RL baselines, highlighting its effectiveness in cross-environment generalization and efficient meta-policy reuse.