Lune

EMNLP2023Top-tier venue

ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer

Huadai Liu, Rongjie Huang, Xuan Lin, Wenqiang Xu, Maozong Zheng, Hong Chen, Jinzheng He, Zhou Zhao

Abstract

Text-to-speech(TTS) has undergone remarkable improvements in performance, particularly with the advent of Denoising Diffusion Probabilistic Models (DDPMs). However, the perceived quality of audio depends not solely on its content, pitch, rhythm, and energy, but also on the physical environment. In this work, we propose ViT-TTS, the first visual TTS model with scalable diffusion transformers. ViT-TTS complement the phoneme sequence with the visual information to generate high-perceived audio, opening up new avenues for practical applications of AR and VR to allow a more immersive and realistic audio experience. To mitigate the data scarcity in learning visual acoustic information, we 1) introduce a selfsupervised learning framework to enhance both the visual-text encoder and denoiser decoder; 2) leverage the diffusion transformer scalable in terms of parameters and capacity to learn visual scene information. Experimental results demonstrate that ViT-TTS achieves new stateof-the-art results, outperforming cascaded systems and other baselines regardless of the visibility of the scene. With low-resource data (1h, 2h, 5h), ViT-TTS achieves comparative results with rich-resource baselines. 1 2 1 Audio samples are available at https://ViT-TTS. github.io/ .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

11 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • FlashAudio: Rectified Flow for Fast and High-Fidelity Text-to-Audio Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • FlashAudio: Rectified Flow for Fast and High-Fidelity Text-to-Audio Generation cites Denoising Diffusion Probabilistic Models
  • Adding Conditional Control to Text-to-Image Diffusion Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-Speech cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers11

  1. Learning-to-Cache: Accelerating Diffusion Transformer via Layer CachingXinyin Ma, Gongfan Fang, Michael Bi Mi, Xinchao WangNeurIPS 2024 · 167 citations
  2. FlashAudio: Rectified Flow for Fast and High-Fidelity Text-to-Audio GenerationHuadai Liu, Jialei Wang, Rongjie Huang, Yang Liu et al.ACL 2025 · 16 citations
  3. EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text PromptingGuanrou Yang, Chen Yang, Qian Chen, Ziyang Ma et al.ACM MM 2025 · 16 citations
  4. The Zeno's Paradox of 'Low-Resource' LanguagesHellina Hailu Nigatu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio et al.EMNLP 2024 · 10 citations
  5. Multi-modal and Multi-scale Spatial Environment Understanding for Immersive Visual Text-to-SpeechRui Liu, Shuwei He, Yifan Hu, Haizhou LiAAAI 2025 · 8 citations

Builds on19

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  3. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  4. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  5. Adding Conditional Control to Text-to-Image Diffusion ModelsLvmin Zhang, Anyi Rao, Maneesh AgrawalaICCV 2023 · 6,759 citations

Related papers

  1. ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation AlignmentJun-Hak Yun, Seung-Bin Kim, Seong-Whan LeeACL 2026
  2. Visual Acoustic MatchingChangan Chen, Ruohan Gao, Paul Calamia, Kristen GraumanCVPR 2022 · 42 citations
  3. DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific FactorsKeon Lee, Dong Won Kim, Jaehyeon Kim, Seungjun Chung et al.ICLR 2025
  4. Prosody-TTS: Improving Prosody with Masked Autoencoder and Conditional Diffusion Model For Expressive Text-to-SpeechRongjie Huang, Chunlei Zhang, Yi Ren, Zhou Zhao et al.ACL Findings 2023 · 10 citations
  5. AV-DiT: Taming Image Diffusion Transformers for Efficient Joint Audio and Video GenerationKai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos et al.ACM MM 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines