Hierarchical Goal Abstractions via Learned Subset Relations

In self-supervised goal-conditioned reinforcement learning (RL) without external rewards, goals are typically specified by observations sampled from experience. However, depending on the observation structure, such a fixed representation of goals may be either too concrete (requiring exact pixel-level matches) or too abstract (involving ambiguous observations). Here we propose the construction of hierarchical latent goal spaces that integrate both concrete and abstract goals. To this end, we use an energy function to learn a partially ordered space, in which a subset relation between observations naturally induces a hierarchy from concrete to abstract goals. This representation enables agents to disambiguate specific states while also generalizing to shared concepts. In experiments on navigation and robotic manipulation, agents trained with our hierarchical goal space achieve higher task success and greater generalization to novel tasks compared to agents limited to purely observational goals.