StreamRAG: Enhancing Real-Time Video Understanding with Retrieval Augmentation

Retrieval-Augmented Generation (RAG) has shown considerable promise in offline video comprehension; however, its application to streaming video remains relatively unexplored. Streaming video introduces unique challenges, such as continuous data influx, temporal sensitivity, and stringent latency requirements. Key obstacles in deploying RAG for streaming video include: (1) the necessity for adaptive semantic segmentation to enable real-time boundary detection; (2) the challenge of balancing latency and accuracy in knowledge extraction; and (3) the complexity of handling queries with varying degrees of temporal sensitivity. To address these issues, we present StreamRAG, an innovative framework designed for streaming video question answering. StreamRAG integrates: (1) a Stream Event Segmentation (SES) module that divides video streams into semantically coherent events; (2) a knowledge extraction accelerator that minimizes captioning latency by reusing previously processed tokens; and (3) a query-aware dynamic knowledge injection module that optimizes retrieval based on the temporal sensitivity of queries and similarity scoring. Experimental results demonstrate that StreamRAG significantly enhances the efficiency of real-time video comprehension while maintaining a balance between accuracy and responsiveness.