Lune

ACL Findings2025

CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis

Fuying Wang, Feng Wu, Yihan Tang, Lequan Yu

Abstract

Integrating multimodal Electronic Health Records (EHR) data-such as numerical time series and free-text clinical reports-has great potential in predicting clinical outcomes. However, prior work has primarily focused on capturing temporal interactions within individual samples and fusing multimodal information, overlooking critical temporal patterns across patients. These patterns, such as trends in vital signs like abnormal heart rate or blood pressure, can indicate deteriorating health or an impending critical event. Similarly, clinical notes often contain textual descriptions that reflect these patterns. Identifying corresponding temporal patterns across different modalities is crucial for improving the accuracy of clinical outcome predictions, yet it remains a challenging task. To address this gap, we introduce a Cross-Modal Temporal Pattern Discovery (CTPD) framework, designed to efficiently extract meaningful cross-modal temporal patterns from multimodal EHR data. Our approach introduces shared initial temporal pattern representations which are refined using slot attention to generate temporal semantic embeddings. To ensure rich cross-modal temporal semantics in the learned patterns, we introduce a Temporal Pattern Noise Contrastive Estimation (TP-NCE) loss for crossmodal alignment, along with two reconstruction losses to retain core information of each modality. Evaluations on two clinically critical tasks-48-hour in-hospital mortality and 24-hour phenotype classification-using the MIMIC-III database demonstrate the superiority of our method over existing approaches.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MSGNet: Learning Multi-Scale Inter-series Correlations for Multivariate Time Series Forecasting cites TS2Vec: Towards Universal Representation of Time Series
  • Multi-Time Attention Networks for Irregularly Sampled Time Series cites Set Functions for Time Series
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. HeartLLM: Discretized ECG Tokenization for LLM-Based Diagnostic ReasoningJinning Yang, Wenjie Sun, Wen ShiAAAI 2026

Builds on24

  1. Object-Centric Learning with Slot AttentionFrancesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran et al.NeurIPS 2020 · 1,275 citations
  2. TS2Vec: Towards Universal Representation of Time SeriesZhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang et al.AAAI 2022 · 938 citations
  3. Multi-Time Attention Networks for Irregularly Sampled Time SeriesSatya Narayan Shukla, Benjamin M. MarlinICLR 2021 · 301 citations
  4. MSGNet: Learning Multi-Scale Inter-series Correlations for Multivariate Time Series ForecastingWanlin Cai, Yuxuan Liang, Xianggen Liu, Jianshuai Feng et al.AAAI 2024 · 239 citations
  5. Set Functions for Time SeriesMax Horn, Michael Moor, Christian Bock, Bastian Rieck et al.ICML 2020 · 199 citations

Related papers

  1. P²²N: Patient Multi-view Multi-modal Feature Fusion Networks for Clinical Outcome PredictionYing Zhang, Baohang Zhou, Kehui Song, Xuhui Sui et al.EMNLP Findings 2022 · 9 citations
  2. THCM-CAL: Temporal-Hierarchical Causal Modelling with Conformal Calibration for Clinical Risk PredictionXin Zhang, Qiyu Wei, Yingjie Zhu, Fanyi Wu et al.EMNLP Findings 2025
  3. Improving Medical Predictions by Irregular Multimodal Electronic Health Records ModelingXinlu Zhang, Shiyang Li, Zhiyu Chen, Xifeng Yan et al.ICML 2023 · 54 citations
  4. Causal Representation Learning from Multimodal Clinical Records under Non-Random Modality MissingnessZihan Liang, Ziwen Pan, Ruoxuan XiongEMNLP 2025
  5. Learning Dynamic Representations and Policies from Multimodal Clinical Time-Series with Informative MissingnessZihan Liang, Ziwen Pan, Ruoxuan XiongACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines