GAFormer: Enhancing Timeseries Transformers Through Group-Aware Embeddings

Analyzing multivariate time series is crucial in numerous domains, yet learning robust and generalizable representations within such datasets remains challenging due to complex inter-channel relationships and non-stationary dynamics. In this paper, we introduce a novel approach for learning data-adaptive position embeddings to incorporate learned spatial and temporal structure into transformer architectures. Our framework introduces group tokens and constructs an instancespecific group embedding (GE) layer that assigns input tokens to a select number of learned group tokens, thereby incorporating structural information into the learning process. Building on this, we propose a novel architecture, the Group-Aware Transformer (GAFormer), which integrates both spatial and temporal group embeddings to achieve state-of-the-art performance on various time series classification and regression tasks. Through evaluations on diverse time series datasets, we demonstrate that GE alone can significantly enhance the performance of several backbone models, and that the combination of spatial and temporal group embeddings allows GAFormer to surpass existing baselines. Moreover, our approach effectively discerns latent structures in data without prior knowledge of the spatial ordering of channels, leading to a more explainable decomposition of the spatial and temporal structure underlying complex timeseries datasets. Code is available at https://github.com/nerdslab/GAFormer .