Lune

EMNLP Findings2024

WavLLM: Towards Robust and Adaptive Speech Large Language Model

Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei

Abstract

Recent advancements in large language models (LLMs) have expanded their scope in natural language processing (NLP) to encompass multimodal functions. However, integrating listening capabilities effectively remains a significant challenge for generalization and complex auditory task execution. In this work, we introduce WavLLM, a robust and adaptive speech large language model featuring dual encoders-a Whisper encoder for semantics and a WavLM encoder for speaker characteristics. Within the two-stage curriculum learning framework, WavLLM first builds its foundational capabilities by optimizing on mixed elementary single tasks, followed by advanced multi-task training on more complex tasks combining elementary ones. To enhance the flexibility and adherence to different tasks and instructions, a prompt-aware LoRA weight adapter is introduced in the second advanced multi-task training stage. We validate the proposed model on universal speech benchmarks and also apply it to specialized speech-question-answer (SQA) dataset, and speech Chain-of-Thought (CoT) evaluation set. Experiments demonstrate that the proposed model achieves state-of-the-art performance across a range of speech tasks in the setting of the same model size, exhibiting robust generalization capabilities in executing complex tasks using CoT approach. The codes, models, audio samples, and SQA evaluation set can be accessed at https://github.com/ microsoft/SpeechT5/tree/main/WavLLM.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

22 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of Sound cites LoRA: Low-Rank Adaptation of Large Language Models
  • AudioTrust: Benchmarking The Multifaceted Trustworthiness of Audio Large Language Models cites Robust Speech Recognition via Large-Scale Weak Supervision
  • QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and Descriptions cites Robust Speech Recognition via Large-Scale Weak Supervision
  • Can Speech LLMs Think while Listening? cites Robust Speech Recognition via Large-Scale Weak Supervision
  • Can Speech LLMs Think while Listening? cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction cites LoRA: Low-Rank Adaptation of Large Language Models
  • ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction cites Robust Speech Recognition via Large-Scale Weak Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers22

  1. ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech InteractionShu-Wen Yang, Ming Tu, Ting-Wei Liu, Xinghua Qu et al.ICLR 2026 · 29 citations
  2. Can Speech LLMs Think while Listening?Yi-Jen Shih, Desh Raj, Chunyang Wu, Wei Zhou et al.ICLR 2026 · 24 citations
  3. AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of SoundGijs Wijngaard, Elia Formisano, Michele Esposito, Michel DumontierNeurIPS 2025 · 22 citations
  4. QualiSpeech: A Speech Quality Assessment Dataset with Natural Language Reasoning and DescriptionsSiyin Wang, Wenyi Yu, Xianzhao Chen, Xiaohai Tian et al.ACL 2025 · 20 citations
  5. AudioTrust: Benchmarking The Multifaceted Trustworthiness of Audio Large Language ModelsKai Li, Can Shen, Yile Liu, Jirui Han et al.ICLR 2026 · 17 citations

Builds on9

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  4. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  5. Language Is Not All You Need: Aligning Perception with Language ModelsShaohan Huang, Li Dong, Wenhui Wang, Yaru Hao et al.NeurIPS 2023 · 810 citations

Related papers

  1. S2ST-Omni: Hierarchical Language-Aware SpeechLLM Adaptation for Multilingual Speech-to-Speech TranslationYu Pan, Xiongfei Wu, Yuguang Yang, Jixun Yao et al.ACL Findings 2026
  2. Whisper-UT: A Unified Translation Framework for Speech and TextCihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz et al.EMNLP 2025
  3. Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio EncodersWeiqiao Shan, Yuang Li, Yuhao Zhang, Yingfeng Luo et al.EMNLP 2025 · 1 citation
  4. LaRA: Large Rank Adaptation for Speech and Text Cross-Modal Learning in Large Language ModelsZuhair Hasan Shaik, Pradyoth Hegde, Prashant Bannulmath, Deepak K. T.EMNLP Findings 2024 · 1 citation
  5. Contrastive Learning for Task-Independent SpeechLLM-PretrainingMaike Züfle, Jan NiehuesACL Findings 2025
Dusk over the sea between two cliffs drawn in fine vertical lines