Lune

EMNLP Findings2024

HSDreport: Heart Sound Diagnosis with Echocardiography Reports

Zihan Zhao, Pingjie Wang, Liudan Zhao, Yuchen Yang, Ya Zhang, Kun Sun, Xin Sun, Xin Zhou, Yu Wang, Yanfeng Wang

Abstract

Heart sound auscultation holds significant importance in the diagnosis of congenital heart disease. However, existing methods for Heart Sound Diagnosis (HSD) tasks are predominantly limited to a few fixed categories, framing the HSD task as a rigid classification problem that does not fully align with medical practice and offers only limited information to physicians. Besides, such methods do not utilize echocardiography reports, the gold standard in the diagnosis of related diseases. To tackle this challenge, we introduce HSDreport, a new benchmark for HSD, which mandates the direct utilization of heart sounds obtained from auscultation to predict echocardiography reports. This benchmark aims to merge the convenience of auscultation with the comprehensive nature of echocardiography reports. First, we collect a new dataset for this benchmark, comprising 2,275 heart sound samples along with their corresponding reports. Subsequently, we develop a knowledge-aware query-based transformer to handle this task. The intent is to leverage the capabilities of medically pre-trained models and the internal knowledge of large language models (LLMs) to address the task's inherent complexity and variability, thereby enhancing the robustness and scientific validity of the method. Furthermore, our experimental results indicate that our method significantly outperforms traditional HSD approaches and existing multimodal LLMs in detecting key abnormalities in heart sounds.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Mining Useful General Data for Low-Resource Domain AdaptationPingjie Wang, Hongcheng Liu, Yusheng Liao, Ziqing Fan et al.ICML 2026 · 3 citations

Builds on8

  1. DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingFeng Li, Hao Zhang, Shilong Liu, Jian Guo et al.CVPR 2022 · 879 citations
  2. On the Effects of Transformer Size on In- and Out-of-Domain CalibrationSoham Dan, Dan RothEMNLP Findings 2021 · 13 citations
  3. Rethinking Multi-Modal Alignment in Multi-Choice VideoQA from Feature and Sample PerspectivesShaoning Xiao, Long Chen, Kaifeng Gao, Zhao Wang et al.EMNLP 2022 · 5 citations
  4. Large Language Models and Multimodal Retrieval for Visual Word Sense DisambiguationAnastasia Kritharoula, Maria Lymperaiou, Giorgos StamouEMNLP 2023 · 3 citations
  5. Non-autoregressive Streaming Transformer for Simultaneous TranslationZhengrui Ma, Shaolei Zhang, Shoutao Guo, Chenze Shao et al.EMNLP 2023 · 3 citations

Related papers

  1. HEARTS: Benchmarking LLM Reasoning on Health Time SeriesSirui Li, Shuhan Xiao, Mihir Joshi, Ahmed Metwally et al.ICML 2026 · 7 citations
  2. MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report GenerationZhongwei Wan, Che Liu, Xin Wang, Chaofan Tao et al.ACL Findings 2025 · 16 citations
  3. EchoVLM: Dynamic Mixture-of-Experts Vision-Language Model for Universal Ultrasound IntelligenceChaoyin She, Ruifang Lu, Lida Chen, Wei Wang et al.ACL 2026 · 8 citations
  4. EchoMLLM: Incentivizing Echocardiographic Video Understanding with Keyframe Grounding and Report GenerationHeyu Huang, Wanran Sun, Chi Chen, Bo Chen et al.ACL Findings 2026
  5. Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge EnhancementChe Liu, Zhongwei Wan, Cheng Ouyang, Anand Shah et al.ICML 2024 · 83 citations
Dusk over the sea between two cliffs drawn in fine vertical lines