STIV: Scalable Text and Image Conditioned Video Generation
Zongyu Lin, Wei Liu, Chen Chen, Jiasen Lu, Wenze Hu, Tsu-Jui Fu, Jesse Allardice, Zhengfeng Lai, Liangchen Song, Bowen Zhang, Cha Chen, Yiran Fei
Abstract
The field of video generation has made remarkable advancements, yet there remains a pressing need for a clear, systematic recipe that can guide the development of robust and scalable models. In this work, we present a comprehensive study that systematically explores the interplay of model architectures, training recipes, and data curation strategies, culminating in a simple and scalable text-image-conditioned video generation method, named STIV. Our framework integrates image condition into a Diffusion Transformer (DiT) through frame replacement, while incorporating text conditioning via a joint image-text conditional classifier-free guidance. This design enables STIV to perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks simultaneously. Additionally, STIV can be easily extended to various applications, such as video prediction, frame interpolation, multi-view generation, and long video generation, etc. With comprehensive ablation studies on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple design. An 8.7B model with 512 2 resolution achieves 83.1 on VBench T2V, surpassing both leading open and closed-source models like CogVideoX-5B, Pika, Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result of 90.1 on VBench I2V task at 512 2 resolution. By providing a transparent and extensible recipe for building cutting-edge video generation models, we aim to empower future research and accelerate progress toward more versatile and reliable video generation solutions.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Lumina-Image 2.0: a Unified and Efficient Image Generative Framework cites High-Resolution Image Synthesis with Latent Diffusion Models
- Lumina-Image 2.0: a Unified and Efficient Image Generative Framework cites SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- Lumina-Image 2.0: a Unified and Efficient Image Generative Framework cites Scalable Diffusion Models with Transformers
- Lumina-Image 2.0: a Unified and Efficient Image Generative Framework cites Learning Transferable Visual Models From Natural Language Supervision
- Lumina-Image 2.0: a Unified and Efficient Image Generative Framework cites Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Scalable Diffusion Models with Transformers
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
- Scalable Diffusion Models with Transformers cites High-Resolution Image Synthesis with Latent Diffusion Models
- Scalable Diffusion Models with Transformers cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Diffusion Models with Flexible Representation Guidance cites High-Resolution Image Synthesis with Latent Diffusion Models
- Learning Diffusion Models with Flexible Representation Guidance cites Scalable Diffusion Models with Transformers
- TIDE: Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
- TIDE: Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation cites SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- TIDE: Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation cites Scalable Diffusion Models with Transformers
- TIDE: Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation cites Learning Transferable Visual Models From Natural Language Supervision
- TIDE: Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image Generation cites Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory cites High-Resolution Image Synthesis with Latent Diffusion Models
- OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory cites Scalable Diffusion Models with Transformers
- OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory cites Learning Transferable Visual Models From Natural Language Supervision
- OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory cites Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites Scalable Diffusion Models with Transformers
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers8
- OneStory: Coherent Multi-Shot Video Generation with Adaptive MemoryZhaochong An, Menglin Jia, Haonan Qiu, Zijian Zhou et al.CVPR 2026 · 33 citations
- Learning Diffusion Models with Flexible Representation GuidanceChenyu Wang, Cai Zhou, Sharut Gupta, Johnson Lin et al.NeurIPS 2025 · 12 citations
- Lumina-Image 2.0: a Unified and Efficient Image Generative FrameworkQi Qin, Le Zhuo, Yi Xin, Ruoyi Du et al.ICCV 2025 · 11 citations
- TIDE: Temporal-Aware Sparse Autoencoders for Interpretable Diffusion Transformers in Image GenerationVictor Shea-Jay Huang, Le Zhuo, Yi Xin, Zhaokai Wang et al.AAAI 2026 · 10 citations
- Chain of Event-Centric Causal Thought for Physically Plausible Video GenerationZixuan Wang, Yixin Hu, Haolan Wang, Feng Chen et al.CVPR 2026 · 8 citations
Builds on29
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations
- SDXL: Improving Latent Diffusion Models for High-Resolution Image SynthesisDustin Podell, Zion English, Kyle Lacey, Andreas Blattmann et al.ICLR 2024 · 4,569 citations
- Scaling Rectified Flow Transformers for High-Resolution Image SynthesisPatrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari et al.ICML 2024 · 3,620 citations
Related papers
- ConditionVideo: Training-Free Condition-Guided Video GenerationBo Peng, Xinyuan Chen, Yaohui Wang, Chaochao Lu et al.AAAI 2024 · 7 citations
- Video Diffusion ModelsJonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan et al.NeurIPS 2022 · 2,948 citations
- ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video GenerationMingyang Wu, Ashirbad Mishra, Soumik Dey, Shuo Xing et al.CVPR 2026 · 8 citations
- REGEN: Learning Compact Video Embedding with (Re-)Generative DecoderYitian Zhang, Long Mai, Aniruddha Mahapatra, David Bourgin et al.ICCV 2025
- GenTron: Diffusion Transformers for Image and Video GenerationShoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong et al.CVPR 2024
