OpenScene: 3D Scene Understanding with Open Vocabularies

Figure 1 . Open-vocabulary 3D Scene Understanding. We propose OpenScene, a zero-shot approach to 3D scene understanding that co-embeds dense 3D point features with image pixels and text. The examples above show a 3D scene with surface points colored by how well they match a user-specified query string -yellow is highest, green is middle, blue is low. Because its features are language-based, OpenScene answers a wide variety of example queries, like "soft", "kitchen", or "work", without labeled 3D data.