ACL Findings2025
VideoRAG: Retrieval-Augmented Generation over Video Corpus
Soyeong Jeong, Kangsan Kim, Jinheon Baek, Sung Ju Hwang
Abstract
Retrieval-Augmented Generation (RAG) is a powerful strategy for improving the factual accuracy of models by retrieving external knowledge relevant to queries and incorporating it into the generation process. However, existing approaches primarily focus on text, with some recent advancements considering images, and they largely overlook videos, a rich source of multimodal knowledge capable of representing contextual details more effectively than any other modality. Also, while very recent studies explore the use of videos in response generation, they either predefine query-associated videos without retrieval or convert videos into textual descriptions, losing multimodal richness. To tackle these, we introduce VideoRAG, a novel framework that not only dynamically retrieves videos based on their relevance with queries but also utilizes both visual and textual information. The operation of VideoRAG is powered by recent Large Video Language Models (LVLMs), which enable the direct processing of video content to represent it for retrieval and the seamless integration of retrieved videos jointly with queries for response generation. Also, inspired by that the context size of LVLMs may not be sufficient to process all frames in extremely long videos and not all frames are equally important, we introduce a video frame selection mechanism to extract the most informative subset of frames, along with a strategy to extract textual information from videos (as it can aid the understanding of video content) when their subtitles are not available. We experimentally validate the effectiveness of VideoRAG, showcasing that it is superior to relevant baselines. Our code is available at https://github.com/starsuzi/VideoRAG.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
17 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension cites Robust Speech Recognition via Large-Scale Weak Supervision
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension cites Learning Transferable Visual Models From Natural Language Supervision
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Robust Speech Recognition via Large-Scale Weak Supervision cites Learning Transferable Visual Models From Natural Language Supervision
- WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning cites Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
- ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding cites Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension
- Learning Transferable Visual Models From Natural Language Supervision cites HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understanding cites Robust Speech Recognition via Large-Scale Weak Supervision
Cited by top-tier papers17
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video ComprehensionYongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin et al.NeurIPS 2025 · 164 citations
- ReAgent-V: A Reward-Driven Multi-Agent Framework for Video UnderstandingYiyang Zhou, Yangfan He, Yaofeng Su, Siwei Han et al.NeurIPS 2025 · 55 citations
- WorldMM: Dynamic Multimodal Memory Agent for Long Video ReasoningWoongyeong Yeo, Kangsan Kim, Jaehong Yoon, Sung Ju HwangCVPR 2026 · 53 citations
- MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing AgentsXijia Tao, Yihua Teng, Xinxing Su, Xinyu Fu et al.ICLR 2026 · 37 citations
- VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video UnderstandingYufei Yin, Qianke Meng, Minghao Chen, Jiajun Ding et al.CVPR 2026 · 24 citations
Builds on32
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video ClipsAntoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi et al.ICCV 2019 · 1,437 citations
Related papers
- ViG-RAG: Video-aware Graph Retrieval-Augmented Generation via Temporal and Semantic Hybrid ReasoningZongsheng Cao, Anran Liu, Yangfan He, Jing Li et al.AAAI 2026 · 1 citation
- Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video UnderstandingXiaoqian Shen, Wenxuan Zhang, Jun Chen, Mohamed ElhoseinyNeurIPS 2025 · 37 citations
- AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video UnderstandingZhucun Xue, Jiangning Zhang, Xurong Xie, Yuxuan Cai et al.NeurIPS 2025 · 19 citations
- VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAGHonghao Fu, Miao Xu, Yiwei Wang, Dailing Zhang et al.ACL 2026 · 2 citations
- MR-RAG: Multimodal Relevance-Aware Retrieval-Augmented Generation for Medical Visual Question AnsweringXuze Li, Haozhao Wang, Zhenyu Huang, Zhongxu Wang et al.CVPR 2026
