GIT-BO: High-Dimensional Bayesian Optimization with Tabular Foundation Models

Bayesian optimization (BO) struggles in high dimensions, where Gaussianprocess surrogates demand heavy retraining and brittle assumptions, slowing progress on real engineering and design problems. We introduce GIT-BO, a Gradient-Informed BO framework that couples TabPFN v2, a tabular foundation model (TFM) that performs zero-shot Bayesian inference in context, with an active-subspace mechanism computed from the model's own predictive-mean gradients. This aligns exploration to an intrinsic low-dimensional subspace via a Fisher-information estimate and selects queries with a UCB acquisition, requiring no online retraining. Across 60 problem variants spanning 20 benchmarks-nine scalable synthetic families and eleven real-world tasks (e.g., power systems, Rover, MOPTA08, Mazda)-up to 500 dimensions, GIT-BO delivers a better performance-time trade-off than state-of-the-art GP-based methods (SAASBO, TuRBO, Vanilla BO, BAxUS), ranking highest in performance and with runtime advantages that grow with dimensionality. Limitations include memory footprint and dependence on the capacity of the underlying TFM. Recent advances in tabular foundation models (TFMs) provide a radically different surrogate modeling paradigm. Prior-Data Fitted Networks (PFNs) (Müller et al., 2022; Hollmann et al., 2022; Müller et al., 2023) perform Bayesian inference in-context with frozen weights, eliminating kernel re-fitting and delivering 10-100× speedups on BO tasks (Rakotoarison et al., 2024; Yu et al., 2025). These approaches address computational bottlenecks by leveraging pre-trained models' in-context learning capability, which requires only a single forward pass at inference during optimization. These powerful TFMs trained on millions of synthetic prior data can also perform accurate inference without additional hyperparameter tuning for a new domain.