Efficient Distributed Optimization under Heavy-Tailed Noise

Distributed optimization has become the default training paradigm in modern machine learning due to the growing scale of models and datasets. To mitigate communication overhead, local updates are often applied before global aggregation, resulting in a nested optimization approach with inner and outer steps. However, heavy-tailed stochastic gradient noise remains a significant challenge, particularly in attention-based models, hindering effective training. In this work, we propose TailOPT, an efficient framework designed to address heavy-tailed noise by leveraging adaptive optimization and novel clipping techniques. We establish convergence guarantees for the TailOPT framework under heavy-tailed noise with local updates and potentially unbounded gradient variance. Among its variants, we propose a memoryand communication-efficient instantiation (named Bi 2 Clip) that performs coordinate-wise clipping from both above and below at both the inner and outer optimizers. Bi 2 Clip brings about benefits of adaptive optimization (e.g., Adam) without the cost of maintaining or transmitting additional gradient statistics. Empirically, TailOPT, including Bi 2 Clip, demonstrates superior performance on various tasks and models compared with state-ofthe-art methods, while being more efficient.