OpenESS: Event-Based Semantic Scene Understanding with Open Vocabularies

Zero-Shot Semantic Segmentation "driveable" (Adjective) "walkable" (Adjective) "car" (Fine-Grained) "manmade" (Coarse) "flat" (Coarse) "barrier" (Fine-Grained) Back Build Road Car Pole Veg Wall Figure 1. Open-vocabulary event-based semantic segmentation (OpenESS). Our framework is capable of performing zero-shot semantic segmentation of event data streams with open vocabularies. Given raw events and text prompts as inputs, OpenESS outputs semantically coherent open-world predictions across various adjective, fine-grained, and coarse categories. The last three columns show the languageguided attention maps where regions of a high similarity score to the given text prompts are highlighted. Best viewed in colors.