Training Diffusion Language Models for Black-Box Optimization

We study offline black-box optimization (BBO), aiming to discover improved designs from an offline dataset of designs and labels, a problem common in robotics and DNA with limited labeled samples. While recent work applies autoregressive LLMs to BBO by formatting tasks as naturallanguage prompts, their left-to-right design generation struggles to capture the strong bidirectional dependencies inherent in design problems. To address this, we propose adapting diffusion LLMs to offline BBO to leverage their bidirectional modeling capabilities. However, a domain gap exists between the natural text pre-training of diffusion LLMs and the heterogeneous signals in BBO (prompts, designs, and labels). To bridge this gap, we construct a unified prompt-response corpus and introduce delimiter tokens to explicitly mark field boundaries for domain adaptation. We further propose a two-stage post-training framework to align the diffusion LLM generation with highlabel designs. The first stage performs supervised fine-tuning on the unified dataset via maskedresponse prediction, and the second stage adopts reinforcement learning with rewards defined by label improvements. Our method achieves state-ofthe-art results on Design-Bench under small-data settings with highly efficient training, requiring only 1.5 H100 GPU hours for discrete tasks. Code for our work is available here. * Equal contribution †Project lead. ‡Work done independent of the author's position at Amazon AGI.