SquareχPO: Differentially Private and Robust χ2-Preference Optimization in Offline Direct Alignment

We theoretically study the offline alignment of language models with human preference feedback, under both preference label corruption and privacy protections. To this end, we propose SquareχPO, a simple one-line change to χPO where the standard log-loss is replaced by a new square loss over probability. This allows us to advance the state-of-the-art of differentially private and robust offline direct alignment. Specifically, for the local model of label privacy, SquareχPO is the first algorithm that attains an optimal rate based on single-policy concentrability, even with general function approximations. On the robustness side against Huber label corruption, SquareχPO is the first alignment method that has a meaningful theoretical guarantee under general function approximations. More importantly, SquareχPO can address privacy protection and corruption simultaneously, where an interesting separation is observed, implying that the order of privacy and corruption matters.