On the Second-order Convergence Properties of Random Search Methods

We study the theoretical convergence properties of random-search methods when optimizing non-convex objective functions without having access to derivatives. We prove that standard random-search methods that do not rely on second-order information converge to a second-order stationary point. However, they suffer from an exponential complexity in terms of the input dimension of the problem. In order to address this issue, we propose a novel variant of random search that exploits negative curvature by only relying on function evaluations. We prove that this approach converges to a second-order stationary point at a much faster rate than vanilla methods: namely, the complexity in terms of the number of function evaluations is only linear in the problem dimension. We test our algorithm empirically and find good agreements with our theoretical results. * Alphabetical ordering, all authors contributed equally. 1 In this manuscript, we will use the terms "random direct-search" and "random search" interchangeably. 35th Conference on Neural Information Processing Systems (NeurIPS 2021).