Lune

ACL Findings2023

FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models

Ziyue Jiang, Qian Yang, Jialong Zuo, Zhenhui Ye, Rongjie Huang, Yi Ren, Zhou Zhao

Abstract

Stutter removal is an essential scenario in the field of speech editing. However, when the speech recording contains stutters, the existing text-based speech editing approaches still suffer from: 1) the over-smoothing problem in the edited speech; 2) lack of robustness due to the noise introduced by stutter; 3) to remove the stutters, users are required to determine the edited region manually. To tackle the challenges in stutter removal, we propose Flu-entSpeech, a stutter-oriented automatic speech editing model. Specifically, 1) we propose a context-aware diffusion model that iteratively refines the modified mel-spectrogram with the guidance of context features; 2) we introduce a stutter predictor module to inject the stutter information into the hidden sequence; 3) we also propose a stutter-oriented automatic speech editing (SASE) dataset that contains spontaneous speech recordings with time-aligned stutter labels to train the automatic stutter localization model. Experimental results on VCTK and LibriTTS datasets demonstrate that our model achieves state-of-the-art performance on speech editing. Further experiments on our SASE dataset show that FluentSpeech can effectively improve the fluency of stuttering speech in terms of objective and subjective metrics. Code and audio samples can be found at https://github.com/Zain-Jiang/ Speech-Editing-Toolkit.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech cites Denoising Diffusion Probabilistic Models
  • Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech cites DiffWave: A Versatile Diffusion Model for Audio Synthesis
  • Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching cites HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
  • Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching cites Denoising Diffusion Probabilistic Models
  • Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
  • Diffusion Models Beat GANs on Image Synthesis cites DiffWave: A Versatile Diffusion Model for Audio Synthesis
  • SongCreator: Lyrics-based Universal Song Generation cites Denoising Diffusion Probabilistic Models
  • DiffWave: A Versatile Diffusion Model for Audio Synthesis cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. SongCreator: Lyrics-based Universal Song GenerationShun Lei, Yixuan Zhou, Boshi Tang, Max W. Y. Lam et al.NeurIPS 2024 · 33 citations
  2. Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow MatchingJialong Zuo, Shengpeng Ji, Minghui Fang, Mingze Li et al.ACL 2025 · 1 citation

Builds on12

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
  3. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech SynthesisJungil Kong, Jaehyeon Kim, Jaekyoung BaeNeurIPS 2020 · 2,890 citations
  4. DiffWave: A Versatile Diffusion Model for Audio SynthesisZhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao et al.ICLR 2021 · 1,902 citations
  5. Grad-TTS: A Diffusion Probabilistic Model for Text-to-SpeechVadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova et al.ICML 2021 · 715 citations

Related papers

  1. UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and VocodingChenpeng Du, Yiwei Guo, Feiyu Shen, Zhijun Liu et al.AAAI 2024 · 64 citations
  2. End-to-End Speech Recognition and Disfluency RemovalParia Jamshid Lou, Mark JohnsonEMNLP Findings 2020 · 20 citations
  3. Prompt-guided Precise Audio Editing with Diffusion ModelsManjie Xu, Chenxing Li, Duzhen Zhang, Dan Su et al.ICML 2024 · 15 citations
  4. Adversarial Training for Low-Resource Disfluency CorrectionVineet Bhat, Preethi Jyothi, Pushpak BhattacharyyaACL Findings 2023 · 3 citations
  5. AUDIT: Audio Editing by Following Instructions with Latent Diffusion ModelsYuancheng Wang, Zeqian Ju, Xu Tan, Lei He et al.NeurIPS 2023 · 120 citations
Dusk over the sea between two cliffs drawn in fine vertical lines