Exploring State-Space Models for Data-Specific Neural Representations

This paper studies the problem of data-specific neural representations, aiming for compact, flexible, and modality-agnostic storage of individual visual data using neural networks. Our approach considers a visual datum as a set of discrete observations of an underlying continuous signal, thus requiring models capable of capturing the inherent structure of the signal. For this purpose, we investigate state-space models (SSMs), which are well-suited for modeling latent signal dynamics. We first explore the appealing properties of SSMs for data-specific neural representation and then present a novel framework that integrates SSMs into the representation pipeline. The proposed framework achieved compact representations and strong reconstruction performance across a range of visual data formats, suggesting the potential of SSMs for data-specific neural representations. Recently, the rise of state-space models (SSMs) has opened a new pathway to this challenge, as SSMs provide a framework for modeling continuous signals in a way that aligns with the objectives of compact neural representations. To be specific, the hidden state of SSM was initially designed to represent the coefficients that reconstruct observed data using a set of orthogonal polynomial bases (Gu et al., 2020; 2022b), which generalizes to the traditional compression algorithms. Although