Unlocking the Power of Differentially Private Zeroth-order Optimization for Fine-tuning LLMs

Differentially private zeroth-order optimization (DPZO in short) has shown promise in fine-tuning large language models (LLMs) while protecting record-level privacy. Compared with classical first-order methods, such as DPSGD, the main difference is that DPZO replaces the exact first-order gradients that are computed via back-propagation with its random zeroth-order approximations that are computed via querying the model's losses. However, DPZO still lags in the resulting model utility compared to existing methods, indicating that further work is needed to fully realize its potential. In this paper, we make a solid step towards designing a better differentially private algorithm for fine-tuning LLMs based on zeroth-order optimization. Our design is centered around the major performance issue of differentially private optimization for large models caused by artificial clipping, which creates biases in the model updates. Using our method called DP-AggZO, we theoretically prove that this issue can be mitigated, leading to an improved convergence rate over the prior DPZO methods and better model utility under the same privacy constraints. We back up our theory with extensive experiments, validating the performance improvement of DP-AggZO. Surprisingly, our DP-AggZO even outperforms the state-of-the-art method DP-AdamW significantly on some benchmark settings.