DiffPO: Diffusion-styled Preference Optimization for Inference Time Alignment of Large Language Models

Inference-time alignment provides an efficient alternative for aligning LLMs with humans. However, these approaches still face challenges, such as limited scalability due to policy-specific value functions and latency during the inference phase. In this paper, we propose a novel approach, Diffusion-styled Preference Optimization (DIFFPO), which provides an efficient and policy-agnostic solution for aligning LLMs with humans. By directly performing alignment at sentence level, DIFFPO avoids the time latency associated with token-level generation. Designed as a plug-and-play module, DIFFPO can be seamlessly integrated with various base models to enhance their alignment. Extensive experiments on AlpacaEval 2, MT-bench, and HH-RLHF demonstrate that DIFFPO achieves superior alignment performance across various settings, achieving a favorable trade-off between alignment quality and inference-time latency. Furthermore, DIFFPO demonstrates model-agnostic scalability, significantly improving the performance of large models such as Llama-3-70B. Our model and code are available here.