Streaming Videollms for Real-Time Procedural Video Understanding

dibschat.github.io/ProVideLLM Figure 1. Overview of ProVideLLM. A streaming video large language model for real-time procedural video tasks with a low memory footprint. It features a multimodal interleaved cache that stores textual descriptions of long-term observations (spanning several minutes) and visual tokens representing short-term observations (spanning a few seconds). We also introduce a new DETR-QFormer connector for better fine-grained tokenization of the short-term. ProVideLLM is capable of handling multiple procedural tasks within a single model.