MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control
Abstract
MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference—removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody—while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba--TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel--diffusion--vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba--attention hybrids in MOS/CMOS, F RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by . Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability. Code available at: https://github.com/sahilkumar15/MVC.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models cites Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models cites ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models cites Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
- ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech cites Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
- ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech cites FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
- ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech cites Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
- Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech cites FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
Builds on6
- Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-SpeechJaehyeon Kim, Jungil Kong, Juhee SonICML 2021 · 1,267 citations
- Grad-TTS: A Diffusion Probabilistic Model for Text-to-SpeechVadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova et al.ICML 2021 · 715 citations
- FastSpeech 2: Fast and High-Quality End-to-End Text to SpeechYi Ren, Chenxu Hu, Xu Tan, Tao Qin et al.ICLR 2021 · 513 citations
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language ModelsYinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler et al.NeurIPS 2023 · 324 citations
- ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-SpeechRongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu et al.ACM MM 2022 · 182 citations
Related papers
- DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech SynthesisYinghao Aaron Li, Rithesh Kumar, Zeyu JinICML 2025
- DiffuMamba: High-Throughput Diffusion LMs with Mamba BackboneVaibhav Singh, Oleksiy Ostapenko, Pierre-André Noël, Eugene Belilovsky et al.ICML 2026
- Mamba-3: Improved Sequence Modeling using State Space PrinciplesAakash Sunil Lahoti, Kevin Y. Li, Berlin Chen, Caitlin Wang et al.ICLR 2026 · 96 citations
- DisCo_Speech: Controllable Zero-Shot Speech Generation with A Disentangled Speech CodecTao Li, Wenshuo Ge, Zhichao Wang, Zihao Cui et al.ACL 2026 · 1 citation
- StableVC: Style Controllable Zero-Shot Voice Conversion with Conditional Flow MatchingJixun Yao, Yuguang Yang, Yu Pan, Ziqian Ning et al.AAAI 2025 · 13 citations
