Adaptive gradient descent on Riemannian manifolds and its applications to Gaussian variational inference

We propose RAdaGD, a novel family of adaptive gradient descent methods on general Riemannian manifolds. RAdaGD adapts the step size parameter without line search, and includes instances that achieve a non-ergodic convergence guarantee, f (x k ) -f (x ⋆ ) ≤ O(1/k), under local geodesic smoothness and generalized geodesic convexity. A core application of RAdaGD is Gaussian Variational Inference, where our method provides the first convergence guarantee in the absence of L-smoothness of the target log-density, under additional technical assumptions. We also investigate the empirical performance of RAdaGD in numerical simulations and demonstrate its competitiveness in comparison to existing algorithms. * Equal contribution, alphabetically ordered. 1 0 ∥γ ′ (t)∥ , dt is called a minimizing geodesic. The exponential map exp x : T x M → M is defined by exp x (v) = γ(1), where γ(0) = x and γ ′ (0) = v. Here, T x M is the tangent space at x. We call the locally well-defined inverse the logarithmic map and denote it