Lune

ICLR2025Top-tier venue

VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis

Yumeng Li, William H. Beluch, Margret Keuper, Dan Zhang, Anna Khoreva

Abstract

Despite tremendous progress in the field of text-to-video (T2V) synthesis, opensourced T2V diffusion models struggle to generate longer videos with dynamically varying and evolving content. They tend to synthesize quasi-static videos, ignoring the necessary visual change-over-time implied in the text prompt. At the same time, scaling these models to enable longer, more dynamic video synthesis often remains computationally intractable. To address this challenge, we introduce the concept of Generative Temporal Nursing (GTN), where we aim to alter the generative process on the fly during inference to improve control over the temporal dynamics and enable generation of longer videos. We propose a method for GTN, dubbed VSTAR, which consists of two key ingredients: 1) Video Synopsis Prompting (VSP) -automatic generation of a video synopsis based on the original single prompt leveraging LLMs, which gives accurate textual guidance to different visual states of longer videos, and 2) Temporal Attention Regularization (TAR) -a regularization technique to refine the temporal attention units of the pre-trained T2V diffusion models, which enables control over the video dynamics. We experimentally showcase the superiority of the proposed approach in generating longer, visually appealing videos over existing open-sourced T2V models. We additionally analyze the temporal attention maps realized with and without VSTAR, demonstrating the importance of applying our method to mitigate neglect of the desired visual change over time.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
  • CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites High-Resolution Image Synthesis with Latent Diffusion Models
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
  • Aid: Adapting Image2video Diffusion Models for Instruction-Guided Video Prediction cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Aid: Adapting Image2video Diffusion Models for Instruction-Guided Video Prediction cites AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
  • Aid: Adapting Image2video Diffusion Models for Instruction-Guided Video Prediction cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion ModelsXiaoxue Wu, Bingjie Gao, Yu Qiao, Yaohui Wang et al.ICLR 2026 · 26 citations
  2. Aid: Adapting Image2video Diffusion Models for Instruction-Guided Video PredictionZhen Xing, Qi Dai, Zejia Weng, Zuxuan Wu et al.ICCV 2025 · 4 citations
  3. Ego-InBetween: Generating Object State Transitions in Ego-Centric VideosMengmeng Ge, Takashi Isobe, Xu Jia, Yanan Sun et al.CVPR 2026 · 1 citation

Builds on19

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  4. SDXL: Improving Latent Diffusion Models for High-Resolution Image SynthesisDustin Podell, Zion English, Kyle Lacey, Andreas Blattmann et al.ICLR 2024 · 4,569 citations
  5. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific TuningYuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang et al.ICLR 2024 · 1,493 citations

Related papers

  1. ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video GenerationZongyi Li, Shujie Hu, Shujie Liu, Long Zhou et al.ICLR 2025
  2. LLM-grounded Video Diffusion ModelsLong Lian, Baifeng Shi, Adam Yala, Trevor Darrell et al.ICLR 2024 · 87 citations
  3. InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual GenerationJinlai Liu, Jian Han, Bin Yan, Hui Wu et al.NeurIPS 2025 · 45 citations
  4. Modular-Cam: Modular Dynamic Camera-view Video Generation with LLMZirui Pan, Xin Wang, Yipeng Zhang, Hong Chen et al.AAAI 2025 · 6 citations
  5. Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video GeneratorsLevon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel et al.ICCV 2023 · 800 citations
Dusk over the sea between two cliffs drawn in fine vertical lines