The Statistical Cost of Robust Kernel Hyperparameter Turning

This paper studies the statistical complexity of kernel hyperparameter tuning in the setting of active regression under adversarial noise. We consider the problem of finding the best interpolant from a class of kernels with unknown hyperparameters, assuming only that the noise is square-integrable. We provide finite-sample guarantees for the problem, characterizing how increasing the complexity of the kernel class increases the complexity of learning kernel hyperparameters. For common kernel classes (e.g. squared-exponential kernels with unknown lengthscale), our results show that hyperparameter optimization increases sample complexity by just a logarithmic factor, in comparison to the setting where optimal parameters are known in advance. Our result is based on a subsampling guarantee for linear regression under multiple design matrices which may be of independent interest. Here T dt is the natural 2 norm on [0, T ]. Defined formally in Section 3, Energy µ (y) is a natural measure of the cost of representing the ground truth signal y with the kernel k µ . It is roughly equal to the smallest norm of a signal capable of using k µ to exactly reconstruct y. Intuitively, if the kernel k µ cannot represent y easily, then the associated term Energy µ (y) is large, and hence the interpolation error may be large.