Hierarchical Policy Learning via Spectral Decomposition

In this paper, we identify a semantic decomposition in robot action sequences, separating task-level motion intent from execution-level refinements. By analyzing actions in the spectral domain using the discrete cosine transform (DCT), we observe that low-frequency components capture global motion trajectories, while high-frequency components encode precise timing, alignment, and contact behaviors. Motivated by this structure, we propose Causal Spectral Policy (CSP), which models action generation as a causal coarse-to-fine process: coarse motion is predicted from observation and language, and fine corrections are generated conditionally on the realized trajectory. Across simulation and realworld evaluations, CSP consistently outperforms strong baselines on precision-sensitive manipulation tasks. Additionally, we propose humaninspired teleoperation noise injection as a data augmentation method under which our approach demonstrates strong robustness to noisy demonstrations. https://causal-spectral-policies.github.io/ .