Streaming Videollms for Real-Time Procedural Video Understanding

Dibyadip Chatterjee; Edoardo Remelli; Yale Song; Bugra Tekin; Abhay Mittal; Bharat Bhatnagar; Necati Cihan Camgöz; Shreyas Hampali; Eric Sauser; Shugao Ma; Angela Yao; Fadime Sener

doi:10.1109/iccv51701.2025.02097

ICCV2025顶会

Streaming Videollms for Real-Time Procedural Video Understanding

Dibyadip Chatterjee, Edoardo Remelli, Yale Song, Bugra Tekin, Abhay Mittal, Bharat Bhatnagar, Necati Cihan Camgöz, Shreyas Hampali, Eric Sauser, Shugao Ma, Angela Yao, Fadime Sener

DOI 出版方

2025年份

2被引次数

1顶会引用

摘要

dibschat.github.io/ProVideLLM Figure 1. Overview of ProVideLLM. A streaming video large language model for real-time procedural video tasks with a low memory footprint. It features a multimodal interleaved cache that stores textual descriptions of long-term observations (spanning several minutes) and visual tokens representing short-term observations (spanning a few seconds). We also introduce a new DETR-QFormer connector for better fine-grained tokenization of the short-term. ProVideLLM is capable of handling multiple procedural tasks within a single model.