Lune

ACL Findings2026

Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity

Yucheng Zhou, Jianbing Shen

Abstract

Autoregressive models have shown superior performance and efficiency in image generation, but remain constrained by high computational costs and prolonged training times in video generation. In this study, we explore methods to accelerate training for autoregressive video generation models through empirical analyses. Our results reveal that while training on fewer video frames significantly reduces training time, it also exacerbates error accumulation and introduces inconsistencies in the generated videos. To address these issues, we propose a Local Optimization (Local Opt.) method, which optimizes tokens within localized windows while leveraging contextual information to reduce error propagation. Inspired by Lipschitz continuity, we propose a Representation Continuity (ReCo) strategy to improve the consistency of generated videos. ReCo utilizes continuity loss to constrain representation changes, improving model robustness and reducing error accumulation. Extensive experiments on class-and text-to-video datasets demonstrate that our approach achieves superior performance to the baseline while halving the training cost without sacrificing quality.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers cites Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks
  • Phenaki: Variable Length Video Generation from Open Domain Textual Descriptions cites CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
  • OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation cites CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers
  • OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation cites Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks
  • OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generation cites Phenaki: Variable Length Video Generation from Open Domain Textual Descriptions
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Autoregressive Image Generation without Vector QuantizationTianhong Li, Yonglong Tian, He Li, Mingyang Deng et al.NeurIPS 2024 · 758 citations
  2. Generating Videos with Dynamics-aware Implicit Generative Adversarial NetworksSihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim et al.ICLR 2022 · 227 citations
  3. OmniTokenizer: A Joint Image-Video Tokenizer for Visual GenerationJunke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng et al.NeurIPS 2024 · 132 citations
  4. CogVideo: Large-scale Pretraining for Text-to-Video Generation via TransformersWenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu et al.ICLR 2023 · 116 citations
  5. Phenaki: Variable Length Video Generation from Open Domain Textual DescriptionsRuben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo et al.ICLR 2023 · 79 citations

Related papers

  1. VideoMAR: Autoregressive Video Generation with Continuous TokensHu Yu, Biao Gong, Hangjie Yuan, Dandan Zheng et al.NeurIPS 2025 · 1 citation
  2. EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive GenerationTianwei Xiong, Jun Hao Liew, Zilong Huang, Zhijie Lin et al.CVPR 2026 · 8 citations
  3. Less Is More: Vision Representation Compression for Efficient Video Generation with Large Language ModelsYucheng Zhou, Jihai Zhang, Guanjie Chen, Jianbing Shen et al.AAAI 2026
  4. Autoregressive Video Generation without Vector QuantizationHaoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo et al.ICLR 2025
  5. LARP: Tokenizing Videos with a Learned Autoregressive Generative PriorHanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines