ENTL: Embodied Navigation Trajectory Learner

Note: During the process of preparing and optimizing our code for the final release, we observed a high degree of sensitivity to the dataset order and training regime. As a result, we are unable to fully support the claims made in the paper. Therefore, we have decided to withdraw this paper. We propose Embodied Navigation Trajectory Learner (ENTL), a method for extracting long sequence representations for embodied navigation. Our approach unifies world modeling, localization and imitation learning into a single sequence prediction task. We train our model using vectorquantized predictions of future states conditioned on current states and actions. ENTL's generic architecture enables the sharing of the the spatio-temporal sequence encoder for multiple challenging embodied tasks. We achieve competitive performance on navigation tasks using significantly less data than strong baselines while performing auxiliary tasks such as localization and future frame prediction (a proxy for world modeling). A key property of our approach is that the model is pre-trained without any explicit reward signal, which makes the resulting model generalizable to multiple tasks and environments.