Lune

ACL Findings2025

Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis

Yifan Hu, Rui Liu, Yi Ren, Xiang Yin, Haizhou Li

Abstract

Conversational Speech Synthesis (CSS) aims to align synthesized speech with the emotional and stylistic context of user-agent interactions to achieve empathy. Current generative CSS models face interpretability limitations due to insufficient emotional perception and redundant discrete speech coding. To address the above issues, we present Chain-Talker, a three-stage framework mimicking human cognition: Emotion Understanding derives context-aware emotion descriptors from dialogue history; Semantic Understanding generates compact semantic codes via serialized prediction; and Empathetic Rendering synthesizes expressive speech by integrating both components. To support emotion modeling, we develop CSS-EmCap, an LLM-driven automated pipeline for generating precise conversational speech emotion captions. Experiments on three benchmark datasets demonstrate that Chain-Talker produces more expressive and empathetic speech than existing methods, with CSS-EmCap contributing to reliable emotion modeling. The code and demos are available at: https://github.com/AI-S2-Lab/ Chain-Talker.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation cites SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
  • Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM cites SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
  • Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLM cites HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response GenerationZhuoyue Gao, Xiaohui Wang, Xiaocui Yang, Wen Zhang et al.ACL 2026

Builds on14

  1. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech SynthesisJungil Kong, Jaehyeon Kim, Jaekyoung BaeNeurIPS 2020 · 2,890 citations
  2. Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-SpeechJaehyeon Kim, Jungil Kong, Juhee SonICML 2021 · 1,267 citations
  3. DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language ModelsGe Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou et al.NeurIPS 2023 · 252 citations
  4. SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational AbilitiesDong Zhang, Shimin Li, Xin Zhang, Jun Zhan et al.EMNLP Findings 2023 · 189 citations
  5. Spoken Question Answering and Speech Continuation Using Spectrogram-Powered LLMEliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar et al.ICLR 2024 · 95 citations

Related papers

  1. Generative Expressive Conversational Speech SynthesisRui Liu, Yifan Hu, Yi Ren, Xiang Yin et al.ACM MM 2024 · 15 citations
  2. UniTalker: Conversational Speech-Visual SynthesisYifan Hu, Rui Liu, Yi Ren, Xiang Yin et al.ACM MM 2025 · 2 citations
  3. Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context ModelingRui Liu, Yifan Hu, Yi Ren, Xiang Yin et al.AAAI 2024 · 31 citations
  4. Emotionally Situated Text-to-Speech Synthesis in User-Agent ConversationYuchen Liu, Haoyu Zhang, Shichao Liu, Xiang Yin et al.ACM MM 2023 · 6 citations
  5. CMCU-CSS: Enhancing Naturalness via Commonsense-based Multi-modal Context Understanding in Conversational Speech SynthesisYayue Deng, Jinlong Xue, Fengping Wang, Yingming Gao et al.ACM MM 2023 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines