Lune

CVPR2024Top-tier venue

GenTron: Diffusion Transformers for Image and Video Generation

Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, Juan-Manuel Pérez-Rúa

Abstract

In this study, we explore Transformer-based diffusion models for image and video generation. Despite the dominance of Transformer architectures in various fields due to their flexibility and scalability, the visual generative domain primarily utilizes CNN-based U-Net architectures, particularly in diffusion-based models. We introduce GenTron, a family of Generative models employing Transformer-based diffusion, to address this gap. Our initial step was to adapt Diffusion Transformers (DiTs) from class to text conditioning, a process involving thorough empirical exploration of the conditioning mechanism. We then scale GenTron from approximately 900M to over 3B parameters, observing improvements in visual quality. Furthermore, we extend GenTron to text-to-video generation, incorporating novel motion-free guidance to enhance video quality. In human evaluations against SDXL, GenTron achieves a 51.1% win rate in visual quality (with a 19.8% draw rate), and a 42.3% win rate in text alignment (with a 42.9% draw rate). GenTron notably performs well in T2I-CompBench, highlighting its compositional generation ability. We hope GenTron could provide meaningful insights and serve as a valuable reference for future research. Please refer to the website11https://www.shoufachen.com/gentron_website/ and the arXiv version for the most up-to-date results: https://arxiv.org/abs/2312.04557.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

40 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • ReSim: Reliable World Simulation for Autonomous Driving cites High-Resolution Image Synthesis with Latent Diffusion Models
  • ReSim: Reliable World Simulation for Autonomous Driving cites Learning Transferable Visual Models From Natural Language Supervision
  • ReSim: Reliable World Simulation for Autonomous Driving cites Denoising Diffusion Probabilistic Models
  • Instilling an Active Mind in Avatars via Cognitive Simulation cites Denoising Diffusion Probabilistic Models
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Learning Transferable Visual Models From Natural Language Supervision
  • Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearning cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearning cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Star: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Star: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Zero-Shot Reference-to-Video Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers40

  1. ReSim: Reliable World Simulation for Autonomous DrivingJiazhi Yang, Kashyap Chitta, Shenyuan Gao, Long Chen et al.NeurIPS 2025 · 53 citations
  2. Instilling an Active Mind in Avatars via Cognitive SimulationJianwen Jiang, Weihong Zeng, Zerong Zheng, Jiaqi Yang et al.ICLR 2026 · 26 citations
  3. Star: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-ResolutionRui Xie, Yinhong Liu, Penghao Zhou, Chen Zhao et al.ICCV 2025 · 11 citations
  4. Scaling Zero-Shot Reference-to-Video GenerationZijian Zhou, Shikun Liu, Haozhe Liu, Haonan Qiu et al.CVPR 2026 · 10 citations
  5. Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model UnlearningRenyang Liu, Guanlin Li, Tianwei Zhang, See-Kiong NgICLR 2026 · 9 citations

Builds on32

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  3. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  4. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  5. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations

Related papers

  1. STIV: Scalable Text and Image Conditioned Video GenerationZongyu Lin, Wei Liu, Chen Chen, Jiasen Lu et al.ICCV 2025 · 1 citation
  2. ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video GenerationZongyi Li, Shujie Hu, Shujie Liu, Long Zhou et al.ICLR 2025
  3. REGEN: Learning Compact Video Embedding with (Re-)Generative DecoderYitian Zhang, Long Mai, Aniruddha Mahapatra, David Bourgin et al.ICCV 2025
  4. VDT: General-purpose Video Diffusion Transformers via Mask ModelingHaoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo et al.ICLR 2024 · 117 citations
  5. All are Worth Words: A ViT Backbone for Diffusion ModelsFan Bao, Shen Nie, Kaiwen Xue, Yue Cao et al.CVPR 2023
Dusk over the sea between two cliffs drawn in fine vertical lines