Parameterizing Non-Parametric Meta-Reinforcement Learning Tasks via Subtask Decomposition

Meta-reinforcement learning (meta-RL) techniques have demonstrated remarkable success in generalizing deep reinforcement learning across a range of tasks. Nevertheless, these methods often struggle to generalize beyond tasks with parametric variations. To overcome this challenge, we propose Subtask Decomposition and Virtual Training (SDVT), a novel meta-RL approach that decomposes each nonparametric task into a collection of elementary subtasks and parameterizes the task based on its decomposition. We employ a Gaussian mixture VAE to meta-learn the decomposition process, enabling the agent to reuse policies acquired from common subtasks. Additionally, we propose a virtual training procedure, specifically designed for non-parametric task variability, which generates hypothetical subtask compositions, thereby enhancing generalization to previously unseen subtask compositions. Our method significantly improves performance on the Meta-World ML-10 and ML-45 benchmarks, surpassing current state-of-the-art techniques. * Corresponding author 37th Conference on Neural Information Processing Systems (NeurIPS 2023). (a) "Pick-place" in standard indistribution meta-RL setup. (b) "Pick-place" in out-ofdistribution meta-RL setup. (c) Non-parametric task variation between "Pick-place" and "Sweep-into.