EMNLP Findings2025
VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Format
Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, Dongyan Zhao
Abstract
Recent researches on video large language models (VideoLLM) predominantly focus on model architectures and training datasets, leaving the interaction format between the user and the model under-explored. In existing works, users often interact with VideoLLMs by using the entire video and a query as input, after which the model generates a response. This interaction format constrains the application of Vide-oLLMs in scenarios such as live-streaming comprehension where videos do not end and responses are required in a real-time manner, and also results in unsatisfactory performance on time-sensitive tasks that requires localizing video segments. In this paper, we focus on a video-text duet interaction format. This interaction format is characterized by the continuous playback of the video, and both the user and the model can insert their text messages at any position during the video playback. When a text message ends, the video continues to play, akin to the alternative of two performers in a duet. We construct MMDuetIT, a video-text training dataset designed to adapt VideoLLMs to videotext duet interaction format. We also introduce the Multi-Answer Grounded Video Question Answering (MAGQA) task to benchmark the real-time response ability of VideoLLMs. Trained on MMDuetIT, MMDuet demonstrates that adopting the video-text duet interaction format enables the model to achieve significant improvements in various time-sensitive tasks (76% CIDEr on YouCook2 dense video captioning, 90% mAP on QVHighlights highlight detection and 25% R@0.5 on Charades-STA temporal video grounding) with minimal training efforts, and also enable VideoLLMs to reply in a real-time manner as the video plays.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
18 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant cites TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant cites ViSpeak: Visual Instruction Feedback in Streaming Videos
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant cites Sigmoid Loss for Language Image Pre-Training
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant cites Learning Transferable Visual Models From Natural Language Supervision
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding cites LoRA: Low-Rank Adaptation of Large Language Models
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding cites Learning Transferable Visual Models From Natural Language Supervision
- StreamForest: Efficient Online Video Understanding with Persistent Event Memory cites TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
- StreamForest: Efficient Online Video Understanding with Persistent Event Memory cites Sigmoid Loss for Language Image Pre-Training
- Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video cites LoRA: Low-Rank Adaptation of Large Language Models
- Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video cites TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding
- Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video cites Sigmoid Loss for Language Image Pre-Training
- Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video cites Learning Transferable Visual Models From Natural Language Supervision
- Streaming Video Instruction Tuning cites StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant
- Streaming Video Instruction Tuning cites ViSpeak: Visual Instruction Feedback in Streaming Videos
- Sigmoid Loss for Language Image Pre-Training cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers18
- StreamForest: Efficient Online Video Understanding with Persistent Event MemoryXiangyu Zeng, Kefan Qiu, Qingyu Zhang, Xinhao Li et al.NeurIPS 2025 · 79 citations
- StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming AssistantHaibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu et al.NeurIPS 2025 · 63 citations
- ViSpeak: Visual Instruction Feedback in Streaming VideosShenghao Fu, Qize Yang, Yuan-Ming Li, Yi-Xing Peng et al.ICCV 2025 · 42 citations
- Eyes Wide Open: Ego Proactive Video-LLM for Streaming VideoXueyang Yu, Cheng Shi, Yang Wang, Sibei YangNeurIPS 2025 · 34 citations
- Streaming Video Instruction TuningJiaer Xia, Peixian Chen, Mengdan Zhang, Xing Sun et al.CVPR 2026 · 28 citations
Builds on11
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 2,932 citations
- Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text GenerationJin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai et al.NeurIPS 2022 · 135 citations
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video UnderstandingShuhuai Ren, Linli Yao, Shicheng Li, Xu Sun et al.CVPR 2024 · 83 citations
Related papers
- MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement LearningYueqian Wang, Songxiang Liu, Disong Wang, Nuo Xu et al.ICLR 2026 · 21 citations
- ViLL-E: Video LLM Embeddings for RetrievalRohit Gupta, Jayakrishnan Unnikrishnan, Fan Fei, Sheng Liu et al.ACL 2026
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in VideosShehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao et al.CVPR 2025
- Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language ModelsHaibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao et al.EMNLP Findings 2025 · 14 citations
- RIVER: A Real-Time Interaction Benchmark for Video LLMsYansong Shi, Qingsong Zhao, Tianxiang Jiang, Xiangyu Zeng et al.ICLR 2026 · 12 citations
