Neurons as Detectors of Coherent Sets in Sensory Dynamics

We model sensory streams as observations from high-dimensional stochastic dynamical systems and conceptualize sensory neurons as self-supervised learners of compact representations of such dynamics. From prior experience, neurons learn coherent sets-regions of stimulus state space whose trajectories evolve cohesively over finite times-and assign membership indices to new stimuli. Coherent sets are identified via spectral clustering of the stochastic Koopman operator (SKO), where the sign pattern of a subdominant singular function partitions the state space into minimally coupled regions. For multivariate Ornstein-Uhlenbeck processes, this singular function reduces to a linear projection onto the dominant singular vector of the whitened state-transition matrix. Encoding this singular vector as a receptive field enables neurons to compute membership indices via the projection sign in a biologically plausible manner. Each neuron detects either a predictive coherent set (stimuli with common futures) or a retrospective coherent set (stimuli with common pasts), suggesting a functional dichotomy among neurons. Since neurons lack access to explicit dynamical equations, the requisite singular vectors must be estimated directly from data, for example, via past-future canonical correlation analysis on lag-vector representations-an approach that naturally extends to nonlinear dynamics. This framework provides a novel account of neuronal temporal filtering, the ubiquity of rectification in neural responses, and known functional dichotomies. Coherent-set clustering thus emerges as a fundamental computation underlying sensory processing and transferable to bio-inspired artificial systems. Neurons in early sensory areas are traditionally thought to extract from recent inputs low-dimensional latent variables that are maximally informative about the near future [1, 2, 3, 4]. Such extraction exploits statistical regularities acquired over evolutionary, developmental, and behavioral timescales from previously encountered natural stimuli [5, 6, 7]. To formalize this intuition for temporally correlated sensory stimuli, we postulate that they are generated by high-dimensional, potentially nonlinear, stochastic dynamical processes, and conceptualize neurons as self-supervised learners of coherent sets-regions of the stimulus state space that evolve cohesively over finite time intervals [8, 9]-thus enabling compact representations of sensory dynamics. Coherent sets can be uncovered via spectral clustering of the stochastic Koopman operator (SKO)-a linear, albeit infinite-dimensional, operator that evolves observables over a finite time interval [9, 10]. The sign of the first non-trivial (subdominant) singular function of the SKO partitions state space into two minimally interacting coherent sets (Fig. 1a ). Accordingly, a neuron can compute a membership index of a new input by evaluating the sign of a subdominant singular function. Because singular values and functions remain real even for irreversible dynamics, this approach generalizes metastable set detection beyond the reversible cases that eigenfunction methods require [11] . 39th Conference on Neural Information Processing Systems (NeurIPS 2025).