UniVideo: Unified Understanding, Generation, and Editing for Videos
Cong Wei, Quande Liu, Zixuan Ye, Qiulin Wang, Xintao Wang, Pengfei Wan, Kun Gai, Wenhu Chen
摘要
Unified multimodal models have shown promising results in multimodal content understanding and generation but remain largely limited to the image domain. In this work, we present UniVideo, a versatile framework that extends unified modeling to the video domain. UniVideo adopts a dual-stream design, combining a Multimodal Large Language Model (MLLM) for visual understanding with a Multimodal DiT (MMDiT) for visual generation. This design preserves the MLLM's original text generation capabilities, enables accurate interpretation of complex multimodal instructions, and maintains visual consistency in the generated content. Built on this architecture, UniVideo unifies diverse video generation and editing tasks under a single multimodal instruction paradigm and is jointly trained across them. Extensive experiments demonstrate that UniVideo matches or surpasses state-of-the-art task-specific baselines in visual understanding, text/image-to-video generation, in-context video generation and in-context video editing. Notably, the unified design of UniVideo enables two forms of generalization. First, UniVideo supports task composition, such as combining editing with style transfer, by integrating multiple capabilities within a single instruction. Second, even without explicit training on free-form video editing, UniVideo transfers its editing capability from large-scale image editing data to this setting, handling unseen instructions, such as changing the environment or altering materials within a video. Beyond these core capabilities, UniVideo also supports visual-prompt-based video generation, where the MLLM interprets visual prompts and guides the MMDiT during synthesis. To foster future research, we released our model and code at https://github.com/KlingTeam/UniVideo .
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
引用它的顶会 Paper15
- Scaling Instruction-Based Video Editing with a High-Quality Synthetic DatasetQingyan Bai, Qiuyu Wang, Hao Ouyang, Yue Yu 等CVPR 2026 · 被引用 79 次
- EditVerse: Unifying Image and Video Editing and Generation with In-Context LearningXuan Ju, Tianyu Wang, Yuqian Zhou, He Zhang 等ICLR 2026 · 被引用 56 次
- IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing AssessmentYinan Chen, Jiangning Zhang, Teng Hu, Yuxiang Zeng 等ICLR 2026 · 被引用 29 次
- JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and GenerationKai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu 等NeurIPS 2025 · 被引用 18 次
- VIVA: VLM-Guided Instruction-Based Video Editing with Reward OptimizationXiaoyan Cong, Haotian Yang, Angtian Wang, Yizhi Wang 等CVPR 2026 · 被引用 16 次
它引用的顶会 Paper35
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser 等CVPR 2022 · 被引用 13,123 次
- Adding Conditional Control to Text-to-Image Diffusion ModelsLvmin Zhang, Anyi Rao, Maneesh AgrawalaICCV 2023 · 被引用 6,759 次
- Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray 等ICML 2021 · 被引用 6,356 次
- SDXL: Improving Latent Diffusion Models for High-Resolution Image SynthesisDustin Podell, Zion English, Kyle Lacey, Andreas Blattmann 等ICLR 2024 · 被引用 4,569 次
- Scaling Rectified Flow Transformers for High-Resolution Image SynthesisPatrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari 等ICML 2024 · 被引用 3,620 次
相关 Paper
- Unified In-Context Video EditingZixuan Ye, Xuanhua He, Quande Liu, Qiulin Wang 等ICLR 2026 · 被引用 37 次
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional TokenizationYang Jin, Zhicheng Sun, Kun Xu, Kun Xu 等ICML 2024 · 被引用 94 次
- Realgeneral: Unifying Visual Generation Via Temporal in-Context Learning With Video ModelsYijing Lin, Mengqi Huang, Shuhan Zhuang, Zhendong MaoICCV 2025 · 被引用 1 次
- VILA-U: a Unified Foundation Model Integrating Visual Understanding and GenerationYecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang 等ICLR 2025
- UniVBench: Towards Unified Evaluation for Video Foundation ModelsJianhui Wei, Xiaotian Zhang, Yichen Li, Yuan Wang 等CVPR 2026 · 被引用 12 次
