Fast kernel methods: Sobolev, physics-informed, and additive models

Kernel methods are powerful tools in statistical learning, but their cubic complexity in the sample size $n$ limits their use on large-scale datasets. In this work, we introduce a scalable framework for kernel regression with $\mathcal{O}(n \log n)$ complexity, and designed to fully leverage GPU acceleration. The approach is based on a Fourier representation of kernels combined with non-uniform fast Fourier transforms (NUFFT), enabling exact, fast, and memory-efficient computations. We instantiate our framework in three settings: Sobolev kernel regression, physics-informed regression, and additive models. The proposed estimators are shown to achieve minimax convergence rates, consistent with classical kernel theory. Empirical results demonstrate that our methods can process up to tens of billions of samples within minutes, providing both statistical accuracy and computational scalability. These contributions establish a flexible approach, paving the way for the routine application of kernel methods in large-scale learning tasks, whenever the kernel norm can be efficiently expressed in Fourier space and the ambient dimension $d$ is small. Although the theoretical framework is valid in all dimensions $d$, the fast Sobolev regression package implemented in the paper relies on the CufiNUFFT package, which currently scales exponentially in $d$ and is limited to $d \leq 3$. For similar reasons, the additive model package is only implemented for simple effects without interaction.