Spectral Evolution Search: Efficient Inference-Time Scaling for Reward-Aligned Image Generation

Inference-time scaling offers a flexible way to align visual generative models with downstream objectives without updating model parameters. In modern image generation models, a natural way to do this is to optimize the random noise from which generation starts. However, searching in this high-dimensional noise space is highly inefficient, because many directions have little effect on the final image. We trace this inefficiency to a spectral bias in generative dynamics: model sensitivity to initial perturbations decays rapidly as frequency increases. Based on this insight, we propose Spectral Evolution Search (SES), a plug-and-play framework for initial noise optimization that performs gradient-free evolutionary search in a low-frequency subspace. Theoretically, we derive the Spectral Scaling Prediction from perturbation propagation dynamics, which explains the frequency-dependent impact of perturbations. Extensive experiments across diverse settings show that SES substantially improves the trade-off between generation quality and computational cost, consistently outperforming strong baselines under the same compute budget.