Lune

ICLR2025Top-tier venue

MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation

Trung X. Pham, Tri Ton, Chang D. Yoo

Abstract

We introduce MDSGen, a novel framework for vision-guided open-domain sound generation optimized for model parameter size, memory consumption, and inference speed. This framework incorporates two key innovations: (1) a redundant video feature removal module that filters out unnecessary visual information, and (2) a temporal-aware masking strategy that leverages temporal context for enhanced audio generation accuracy. In contrast to existing resource-heavy Unet-based models, MDSGen employs denoising masked diffusion transformers, facilitating efficient generation without reliance on pre-trained diffusion models. Evaluated on the benchmark VGGSound dataset, our smallest model (5M parameters) achieves 97.9% alignment accuracy, using 172× fewer parameters, 371% less memory, and offering 36× faster inference than the current 860M-parameter state-of-the-art model (93.9% accuracy). The larger model (131M parameters) reaches nearly 99% accuracy while requiring 6.5× fewer parameters. These results highlight the scalability and effectiveness of our approach. The code is available at https://bit.ly/mdsgen.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Adding Conditional Control to Text-to-Image Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Scalable Diffusion Models with Transformers cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation cites Scalable Diffusion Models with Transformers
  • Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation cites Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
  • PAVAS: Physics-Aware Video-to-Audio Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
  • PAVAS: Physics-Aware Video-to-Audio Synthesis cites Scalable Diffusion Models with Transformers
  • PAVAS: Physics-Aware Video-to-Audio Synthesis cites TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-To-Audio Synthesis
  • Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators cites Adding Conditional Control to Text-to-Image Diffusion Models
  • A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers cites High-Resolution Image Synthesis with Latent Diffusion Models
  • A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers cites Scalable Diffusion Models with Transformers
  • Occlusion-Robust Stylization for Drawing-Based 3D Animation cites TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-To-Audio Synthesis
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. PAVAS: Physics-Aware Video-to-Audio SynthesisOh Hyun-Bin, Yuhta Takida, Toshimitsu Uesaka, Tae-Hyun Oh et al.CVPR 2026 · 2 citations
  2. A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion TransformersTrung X. Pham, Kang Zhang, Ji Woo Hong, Chang Dong YooICLR 2026 · 2 citations
  3. Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio GenerationKang Zhang, Trung X. Pham, Suyeon Lee, Axi Niu et al.NeurIPS 2025 · 1 citation
  4. TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-To-Audio SynthesisTri Ton, Ji Woo Hong, Chang D. YooICCV 2025 · 1 citation
  5. Occlusion-Robust Stylization for Drawing-Based 3D AnimationSunjae Yoon, Gwanhyeong Koo, Younghwan Lee, Ji Woo Hong et al.ICCV 2025

Builds on19

  1. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  2. Adding Conditional Control to Text-to-Image Diffusion ModelsLvmin Zhang, Anyi Rao, Maneesh AgrawalaICCV 2023 · 6,759 citations
  3. Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations
  4. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech SynthesisJungil Kong, Jaehyeon Kim, Jaekyoung BaeNeurIPS 2020 · 2,890 citations
  5. Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video GeneratorsLevon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel et al.ICCV 2023 · 800 citations

Related papers

  1. Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow MatchingYongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang et al.NeurIPS 2024 · 73 citations
  2. VAFlow: Video-to-Audio Generation with Cross-Modality Flow MatchingXihua Wang, Xin Cheng, Yuyue Wang, Ruihua Song et al.ICCV 2025 · 6 citations
  3. READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head GenerationHaotian Wang, Yuzhe Weng, Jun Du, Haoran Xu et al.AAAI 2026 · 1 citation
  4. AV-Edit: Multimodal Generative Sound Effect Editing via Audio-Visual Semantic Joint ControlXinyue Guo, Xiaoran Yang, Lipan Zhang, Jianxuan Yang et al.AAAI 2026
  5. SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from VideosAmir Dellali, Luca Lanzendörfer, Florian Grötschla, Roger WattenhoferICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines