A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities

Tabular datasets are inherently heterogeneous, presenting significant challenges for developing pre-trained foundation models. The recently introduced transformerbased Tabular Prior-data Fitted Network v2 (TabPFN v2) achieves unprecedented in-context learning performance across diverse downstream datasets, marking a pivotal advancement in tabular foundation models. In this paper, we take a closer look at TabPFN v2 to examine how it effectively handles heterogeneity and achieves high predictive accuracy, and to explore how its limitations in high-dimensional, many-category, and large-scale tasks can be mitigated. We find that TabPFN v2 can infer attribute relationships even when provided with randomized attribute token inputs, eliminating the need to explicitly learn dataset-specific attribute embeddings to address heterogeneity. We further show that TabPFN v2 can be transformed into a feature extractor, revealing its ability to construct a highly separable feature space for accurate predictions. Lastly, we demonstrate that TabPFN v2's limitations can be addressed through a test-time divide-and-conquer strategy, enabling scalable inference without requiring re-training. By uncovering the mechanisms behind TabPFN v2's success and introducing strategies to extend its applicability, this study offers key insights into the design of future tabular foundation models. Accuracy:0.8600 Accuracy:0.5650 Accuracy:0.8590 Accuracy:0.9070 Accuracy:0.9330 Accuracy:0.9590 Accuracy:0.8876 Accuracy:0.6105 Accuracy:0.8828 Accuracy:0.9046 Accuracy:0.9047 Accuracy:0.9085 Accuracy:0.8782 Accuracy:0.3690 Accuracy:0.7934 Accuracy:0.8598 Accuracy:0.8708 Accuracy:0.9188 Accuracy:0.5899 (a) Raw Feature Accuracy:0.7845