Lune

Abstract

The recent release of GPT-4o showcased the potential of end-to-end multimodal models, not just in terms of low latency but also in their ability to understand and generate expressive speech with rich emotions. While the details are unknown to the open research community, it likely involves significant amounts of curated data and compute, neither of which is readily accessible. In this paper, we present BLSP-Emo (Bootstrapped Language-Speech Pretraining with Emotion support), a novel approach to developing an end-to-end speechlanguage model capable of understanding both semantics and emotions in speech and generate empathetic responses. BLSP-Emo utilizes existing speech recognition (ASR) and speech emotion recognition (SER) datasets through a two-stage process. The first stage focuses on semantic alignment, following recent work on pretraining speech-language models using ASR data. The second stage performs emotion alignment with the pretrained speech-language model on an emotion-aware continuation task constructed from SER data. Our experiments demonstrate that the BLSP-Emo model excels in comprehending speech and delivering empathetic responses, both in instruction-following tasks and conversations. 1 1 Visit https://github.com/cwang621/blsp-emo for code and https://cwang621.github.io/blsp-emo. github.io for demo. That's wonderful news! Congratulations on your upcoming graduation! Graduating is a significant achievement and a major milestone in your life ... Cheerful I'm sorry to hear that you're feeling sad about graduating soon. It's natural to feel a mix of emotions when it comes to leaving behind the ... 🔊🔊 I'm graduating soon. 🔊🔊 I'm graduating soon.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation cites Robust Speech Recognition via Large-Scale Weak Supervision
  • ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation cites SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
  • ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation cites SALMONN: Towards Generic Hearing Abilities for Large Language Models
  • Robust Speech Recognition via Large-Scale Weak Supervision cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoning cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • SALMONN: Towards Generic Hearing Abilities for Large Language Models cites Robust Speech Recognition via Large-Scale Weak Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion ReasoningDingdong WANG, Shujie LIU, Tianhua Zhang, Youjun Chen et al.ICLR 2026 · 22 citations
  2. FeelWave: Enabling Emotion-Aware Voice Interaction through Noise-Robust mmWave Emotion SensingLingyu Wang, You Zuo, Dequan Wang, Chenming He et al.CHI 2026 · 1 citation
  3. ES4R: Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response GenerationZhuoyue Gao, Xiaohui Wang, Xiaocui Yang, Wen Zhang et al.ACL 2026

Builds on7

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  2. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  3. SALMONN: Towards Generic Hearing Abilities for Large Language ModelsChangli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen et al.ICLR 2024 · 557 citations
  4. SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational AbilitiesDong Zhang, Shimin Li, Xin Zhang, Jun Zhan et al.EMNLP Findings 2023 · 189 citations
  5. #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language ModelsKeming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin et al.ICLR 2024 · 119 citations

Related papers

  1. EMOVA: Empowering Language Models to See, Hear and Speak with Vivid EmotionsKai Chen, Yunhao Gou, Runhui Huang, Zhili Liu et al.CVPR 2025
  2. The MERSA Dataset and a Transformer-Based Approach for Speech Emotion RecognitionEnshi Zhang, Rafael Trujillo, Christian PoellabauerACL 2024
  3. Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech SynthesisYifan Hu, Rui Liu, Yi Ren, Xiang Yin et al.ACL Findings 2025 · 1 citation
  4. VAEmo: Efficient Representation Learning for Visual-Audio Emotion With Knowledge InjectionHao Cheng, Zhiwei Zhao, Yichao He, Zhenzhen Hu et al.ACM MM 2025 · 9 citations
  5. Sentiment-Aware Word and Sentence Level Pre-training for Sentiment AnalysisShuai Fan, Chen Lin, Haonan Li, Zhenghao Lin et al.EMNLP 2022 · 20 citations
Dusk over the sea between two cliffs drawn in fine vertical lines