Lune

NeurIPS2023Top-tier venue

Hyper-HMM: aligning human brains and semantic features in a common latent event space

Caroline Lee, Jane Han, Feilong Ma, Jiahui Guo, James V. Haxby, Christopher Baldassano

Abstract

Naturalistic stimuli evoke complex neural responses with spatial and temporal properties that differ across individuals. Current alignment methods focus on either spatial hyperalignment (assuming exact temporal correspondence) or temporal alignment (assuming exact spatial correspondence). Here, we propose a hybrid model, the Hyper-HMM, that simultaneously aligns both temporal and spatial features across brains. The model learns to linearly project voxels to a reduced-dimension latent space, in which timecourses are segmented into corresponding temporal events. This approach allows tracking of each individual’s mental trajectory through an event sequence and for alignment with other feature spaces such as stimulus content. Using an fMRI dataset in which students watch videos of class lectures, we demonstrate that the Hyper-HMM can be used to map all participants and the semantic content of the videos into a common low-dimensional space, and that these mappings generalize to held-out data. Our model provides a new window into individual cognitive dynamics evoked by complex naturalistic stimuli.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Orthogonal Contrastive Learning for Multi-Representation fMRI AnalysisTony YousefnezhadNeurIPS 2025 · 1 citation

Builds on2

  1. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  2. Toward a realistic model of speech processing in the brain with self-supervised learningJuliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec et al.NeurIPS 2022 · 164 citations

Related papers

  1. The Human Brain as a Dynamic Mixture of Expert Models in Video UnderstandingChristina Sartzetaki, Anne Zonneveld, Pablo Oyarzo, Alessandro T. Gifford et al.ICLR 2026 · 4 citations
  2. Multi-modal brain encoding models for multi-modal stimuliSubba Reddy Oota, Khushbu Pahwa, Mounika Marreddy, Maneesh Kumar Singh et al.ICLR 2025
  3. Neural Latent Aligner: Cross-trial Alignment for Learning Representations of Complex, Naturalistic Neural DataCheol Jun Cho, Edward F. Chang, Gopala Krishna AnumanchipalliICML 2023 · 10 citations
  4. Time-Evolving Dynamical System for Learning Latent Representations of Mouse Visual Neural ActivityLiwei Huang, Zhengyu Ma, Liutao Yu, Huihui Zhou et al.NeurIPS 2025
  5. CrossMind-VL: Multi-Subject Mind-to-Video Decoding with Multimodal LLM Semantic GroundingXuanliu Zhu, Yiqiao Chai, Runnan Li, Mingying Lan et al.ACM MM 2025
Dusk over the sea between two cliffs drawn in fine vertical lines