MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation
Abstract
We introduce MDSGen, a novel framework for vision-guided open-domain sound generation optimized for model parameter size, memory consumption, and inference speed. This framework incorporates two key innovations: (1) a redundant video feature removal module that filters out unnecessary visual information, and (2) a temporal-aware masking strategy that leverages temporal context for enhanced audio generation accuracy. In contrast to existing resource-heavy Unet-based models, MDSGen employs denoising masked diffusion transformers, facilitating efficient generation without reliance on pre-trained diffusion models. Evaluated on the benchmark VGGSound dataset, our smallest model (5M parameters) achieves 97.9% alignment accuracy, using 172× fewer parameters, 371% less memory, and offering 36× faster inference than the current 860M-parameter state-of-the-art model (93.9% accuracy). The larger model (131M parameters) reaches nearly 99% accuracy while requiring 6.5× fewer parameters. These results highlight the scalability and effectiveness of our approach. The code is available at https://bit.ly/mdsgen.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Adding Conditional Control to Text-to-Image Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
- Scalable Diffusion Models with Transformers cites High-Resolution Image Synthesis with Latent Diffusion Models
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation cites Scalable Diffusion Models with Transformers
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation cites Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
- PAVAS: Physics-Aware Video-to-Audio Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
- PAVAS: Physics-Aware Video-to-Audio Synthesis cites Scalable Diffusion Models with Transformers
- PAVAS: Physics-Aware Video-to-Audio Synthesis cites TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-To-Audio Synthesis
- Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators cites High-Resolution Image Synthesis with Latent Diffusion Models
- Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators cites Adding Conditional Control to Text-to-Image Diffusion Models
- A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers cites High-Resolution Image Synthesis with Latent Diffusion Models
- A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion Transformers cites Scalable Diffusion Models with Transformers
- Occlusion-Robust Stylization for Drawing-Based 3D Animation cites TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-To-Audio Synthesis
Cited by top-tier papers6
- PAVAS: Physics-Aware Video-to-Audio SynthesisOh Hyun-Bin, Yuhta Takida, Toshimitsu Uesaka, Tae-Hyun Oh et al.CVPR 2026 · 2 citations
- A Hidden Semantic Bottleneck in Conditional Embeddings of Diffusion TransformersTrung X. Pham, Kang Zhang, Ji Woo Hong, Chang Dong YooICLR 2026 · 2 citations
- Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio GenerationKang Zhang, Trung X. Pham, Suyeon Lee, Axi Niu et al.NeurIPS 2025 · 1 citation
- TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-To-Audio SynthesisTri Ton, Ji Woo Hong, Chang D. YooICCV 2025 · 1 citation
- Occlusion-Robust Stylization for Drawing-Based 3D AnimationSunjae Yoon, Gwanhyeong Koo, Younghwan Lee, Ji Woo Hong et al.ICCV 2025
Builds on19
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Adding Conditional Control to Text-to-Image Diffusion ModelsLvmin Zhang, Anyi Rao, Maneesh AgrawalaICCV 2023 · 6,759 citations
- Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations
- HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech SynthesisJungil Kong, Jaehyeon Kim, Jaekyoung BaeNeurIPS 2020 · 2,890 citations
- Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video GeneratorsLevon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel et al.ICCV 2023 · 800 citations
Related papers
- Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow MatchingYongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang et al.NeurIPS 2024 · 73 citations
- VAFlow: Video-to-Audio Generation with Cross-Modality Flow MatchingXihua Wang, Xin Cheng, Yuyue Wang, Ruihua Song et al.ICCV 2025 · 6 citations
- READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head GenerationHaotian Wang, Yuzhe Weng, Jun Du, Haoran Xu et al.AAAI 2026 · 1 citation
- AV-Edit: Multimodal Generative Sound Effect Editing via Audio-Visual Semantic Joint ControlXinyue Guo, Xiaoran Yang, Lipan Zhang, Jianxuan Yang et al.AAAI 2026
- SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from VideosAmir Dellali, Luca Lanzendörfer, Florian Grötschla, Roger WattenhoferICML 2026
