Low-Rank Extragradient Method for Nonsmooth and Low-Rank Matrix Optimization Problems

Low-rank and nonsmooth matrix optimization problems capture many fundamental tasks in statistics and machine learning. While significant progress has been made in recent years in developing efficient methods for smooth low-rank optimization problems that avoid maintaining high-rank matrices and computing expensive high-rank SVDs, advances for nonsmooth problems have been slow paced. In this paper we consider standard convex relaxations for such problems. Mainly, we prove that under a natural generalized strict complementarity condition and under the relatively mild assumption that the nonsmooth objective can be written as a maximum of smooth functions, the extragradient method, when initialized with a "warm-start" point, converges to an optimal solution with rate O(1/t) while requiring only two low-rank SVDs per iteration. We give a precise trade-off between the rank of the SVDs required and the radius of the ball in which we need to initialize the method. We support our theoretical results with empirical experiments on several nonsmooth low-rank matrix recovery tasks, demonstrating that using simple initializations, the extragradient method produces exactly the same iterates when full-rank SVDs are replaced with SVDs of rank that matches the rank of the (low-rank) ground-truth matrix to be recovered. * This version corrects an error in the original paper published in NeurIPS 2021 [22] : while the version [22] provides convergence rates w.r.t. the best iterate (which under the assumptions of the paper is guaranteed to be low-rank), this corrected version provides the same rates but for the ergodic sequence, i.e., the averaged iterate (which, under our assumptions, is the average of low-rank iterates). 1 in [42, 6, 30] and [1] the authors consider SDPs with linear objective function and affine constraints of the form A(X) = b. By incorporating the linear constraints into the objective function via a ℓ2 penalty term of the form λ A(X)b 2, λ > 0, we obtain a nonsmooth objective function.