Lune

EMNLP Findings2022

P²²N: Patient Multi-view Multi-modal Feature Fusion Networks for Clinical Outcome Prediction

Ying Zhang, Baohang Zhou, Kehui Song, Xuhui Sui, Guoqing Zhao, Ning Jiang, Xiaojie Yuan

Abstract

Clinical outcome prediction is critical to the condition prediction of patients and management of hospital capacities. There are two kinds of medical data, including time series signals recorded by various devices and clinical notes in electronic health records (EHR), which are used for two common prediction targets: mortality and length of stay. Traditional methods focused on utilizing time series data but ignored clinical notes. With the development of deep learning, natural language processing (NLP) and multi-modal learning methods are exploited to jointly model the time series and clinical notes with different modals. However, the existing methods failed to fuse the multi-modal features of patients from different views. Therefore, we propose the patient multi-view multi-modal feature fusion networks for clinical outcome prediction. Firstly, from patient inner view, we propose to utilize the co-attention module to enhance the fine-grained feature interaction between time series and clinical notes from each patient. Secondly, the patient outer view is the correlation between patients, which can be reflected by the structural knowledge in clinical notes. We exploit the structural information extracted from clinical notes to construct the patient correlation graph, and fuse patients' multi-modal features by graph neural networks (GNN). The experimental results on MIMIC-III benchmark demonstrate the superiority of our method.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on2

  1. ConCare: Personalized Clinical Feature Embedding via Capturing the Healthcare ContextLiantao Ma, Chaohe Zhang, Yasha Wang, Wenjie Ruan et al.AAAI 2020 · 190 citations
  2. Predicting in-hospital mortality by combining clinical notes with time-series dataIman Deznabi, Mohit Iyyer, Madalina FiterauACL Findings 2021 · 40 citations

Related papers

  1. How to leverage the multimodal EHR data for better medical prediction?Bo Yang, Lijun WuEMNLP 2021 · 22 citations
  2. CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records AnalysisFuying Wang, Feng Wu, Yihan Tang, Lequan YuACL Findings 2025 · 2 citations
  3. GraphCare: Enhancing Healthcare Predictions with Personalized Knowledge GraphsPengcheng Jiang, Cao Xiao, Adam Cross, Jimeng SunICLR 2024 · 77 citations
  4. Improving Medical Predictions by Irregular Multimodal Electronic Health Records ModelingXinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan et al.ICML 2023 · 54 citations
  5. Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative MissingnessZihan Liang, Ziwen Pan, Ruoxuan XiongACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines