FLAME: Fast Long-context Adaptive Memory for Event-based Vision

We propose Fast Long-context Adaptive Memory for Event (FLAME), a novel scalable architecture that combines neuro-inspired feature extraction with robust structured sequence modeling to efficiently process asynchronous and sparse event camera data. As a departure from conventional input encoding methods, FLAME presents Event Attention Layer, a novel feature extractor that leverages neuromorphic dynamics (Leaky Integrate-and-Fire (LIF)) to directly capture multi-timescale features from event streams. The feature extractor integrates with a structured state-space model with a novel Event-Aware HiPPO (EA-HiPPO) mechanism that dynamically adapts memory retention based on inter-event intervals to understand relationship across varying temporal scales and event sequences. A Normal Plus Low Rank (NPLR) decomposition reduces the computational complexity of state update from O(N 2 ) to O(N r), where N represents the dimension of the core state vector and r is the rank of a low-rank component (with r ≪ N ). FLAME demonstrates state-of-the-art accuracy for event-by-event processing on complex event camera datasets. This paper presents FLAME (Fast Long-context Adaptive Memory for Event-based Systems), a novel, computationally efficient, and scalable SSM-based framework for event-by-event processing of large-scale event-based vision data. The FLAME architecture makes following key contributions.

• Novel Event-Driven Input Encoding with Neuromorphic Dynamics: We propose the Event Attention Layer, which utilizes neuromorphic-inspired dynamics (such as Leaky Integrate-and-Fire mechanisms) for direct, multi-timescale feature extraction from raw asynchronous event streams. This approach overcomes limitations of conventional input encoders for SSMs when applied to event data, by inherently processing information eventby-event and preserving crucial temporal precision and sparsity without requiring dense, fixed-size input patches or hand-crafted features.

• Event-Aware HiPPO (EA-HiPPO) for Adaptive Memory: We present a novel adaptation of HiPPO principles where the state-space dynamics are made explicitly sensitive to the precise timing of discrete input events. Unlike standard HiPPO which assumes continuous inputs, EA-HiPPO dynamically modulates memory retention based on inter-event intervals. This allows it to effectively capture diverse temporal patterns in varying timescales and maintain context within sparse and asynchronous event streams.

• Computationally Efficient SSM Framework for Events: We apply Normal-Plus-Low-Rank (NPLR) decomposition within the EA-HiPPO core, which reduces state update com-