Lune

ACL Findings2026

ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding

Tian Xueyun, Wei Li, Bingbing Xu, Heng Dong, Yuanzhuo Wang, Huawei Shen

Abstract

Recent Omni-multimodal Large Language Models show promise in unified audio, vision, and text modeling. However, streaming audiovideo understanding remains challenging, as existing approaches suffer from disjointed capabilities: they typically exhibit incomplete modality support or lack autonomous proactive monitoring. To address this, we present ROMA, a real-time omni-multimodal assistant for unified reactive and proactive interaction. ROMA processes continuous inputs as synchronized multimodal units, aligning dense audio with discrete video frames to handle granularity mismatches. For online decisionmaking, we introduce a lightweight speak head that decouples response initiation from generation to ensure precise triggering without task conflict. We train ROMA with a curated streaming dataset and a two-stage curriculum that progressively optimizes for streaming format adaptation and proactive responsiveness. To standardize the fragmented evaluation landscape, we reorganize diverse benchmarks into a unified suite covering both proactive (alert, narration) and reactive (QA) settings. Extensive experiments across 12 benchmarks demonstrate ROMA achieves state-of-the-art performance on proactive tasks while competitive in reactive settings, validating its robustness in unified real-time omni-multimodal understanding. Our project page is available at here 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant cites Streaming Long Video Understanding with Large Language Models
  • StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant cites InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
  • StreamingVLM: Real-Time Understanding for Infinite Video Streams cites Streaming Long Video Understanding with Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on22

  1. InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and GenerationYi Wang, Yinan He, Yizhuo Li, Kunchang Li et al.ICLR 2024 · 467 citations
  2. Detecting Moments and Highlights in Videos via Natural Language QueriesJie Lei, Tamara L. Berg, Mohit BansalNeurIPS 2021 · 425 citations
  3. Streaming Long Video Understanding with Large Language ModelsRui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang et al.NeurIPS 2024 · 216 citations
  4. StreamingVLM: Real-Time Understanding for Infinite Video StreamsRuyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He et al.ICLR 2026 · 95 citations
  5. StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming AssistantHaibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu et al.NeurIPS 2025 · 63 citations

Related papers

  1. OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video ContextsYuxuan Wang, Yueqian Wang, Bo Chen, Tong Wu et al.CVPR 2025
  2. Streaming Video Instruction TuningJiaer Xia, Peixian Chen, Mengdan Zhang, Xing Sun et al.CVPR 2026 · 28 citations
  3. VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant ModelsPavan Kumar Anasosalu Vasu, Cem Koc, Fartash Faghri, Chun-Liang Li, Bo Feng, Zhengfeng Lai, Meng Cao, Oncel Tuzel, Hadi PouransariCVPR Findings 2026
  4. RIVER: A Real-Time Interaction Benchmark for Video LLMsYansong Shi, Qingsong Zhao, Tianxiang Jiang, Xiangyu Zeng et al.ICLR 2026 · 12 citations
  5. PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile ScenariosXudong LU, Guan Huankang, Yang Bo, Jinpeng Chen et al.ICML 2026 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines