EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Network

Recent foundational models for tabular data, such as TabPFN, excel at adapting to new tasks via in-context learning, but remain constrained to a fixed, pre-defined number of target dimensions-often necessitating costly ensembling strategies. We trace this constraint to a deeper architectural shortcoming: these models lack target equivariance, so that permuting target dimension orderings alters their predictions. This deficiency gives rise to an irreducible "equivariance gap," an error term that introduces instability in predictions. We eliminate this gap by designing a fully target-equivariant architecture-ensuring permutation invariance via equivariant encoders, decoders, and a bi-attention mechanism. Empirical evaluation on standard classification benchmarks shows that, on datasets with more classes than those seen during pre-training, our model matches or surpasses existing methods while incurring lower computational overhead.

However, models such as TabPFN [12] are inherently confined to covariate-target pairs of fixed, predefined dimension, thereby limiting their applicability to datasets that match these specifications. This limitation can be alleviated via additional data pre-processing-e.g., projecting high-dimensional covariates into a lower-dimensional subspace-or by post-processing model predictions,-e.g. employing hierarchical strategies for classification tasks with many classes [31,5]. However, the increased computational burden often offsets some of the benefits offered by in-context learning.