CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation

For robots to be generally useful, they must be able to find arbitrary objects described by people (i.e., be language-driven) even without expensive navigation training on in-domain data (i.e., perform zero-shot inference). We explore these capabilities in a unified setting: languagedriven zero-shot object navigation (L-ZSON). Inspired by the recent success of open-vocabulary models for image classification, we investigate a straightforward framework, CLIP on Wheels (CoW), to adapt open-vocabulary models to this task without fine-tuning. To better evaluate L-ZSON, we introduce the PASTURE benchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects. We conduct an in-depth empirical study by directly deploying 21 CoW baselines across HABITAT, ROBOTHOR, and PASTURE. In total, we evaluate over 90k navigation episodes and find that ( 1) CoW baselines often struggle to leverage language descriptions, but are proficient at finding uncommon objects. (2) A simple CoW, with CLIP-based object localization and classical exploration-and no additional training-matches the navigation efficiency of a state-of-the-art ZSON method trained for 500M steps on HABITAT MP3D data. This same CoW provides a 15.6 percentage point improvement in success over a state-of-the-art ROBOTHOR ZSON model. 1