General and Efficient Steering of Unconditional Diffusion Models

Steering diffusion models toward conditions unseen during training typically requires either retraining with conditional inputs or per-step gradient computations, both of which incur substantial computational overhead. We present Noise-Aligned RFM Steering (NA-RFM), a general recipe for efficiently steering unconditional diffusion models without gradient guidance during inference, enabling fast controllable generation.

The method combines two offline-computed signals: noise alignment, a high-noise correction from PCA statistics of the target examples and the full data, and Recursive Feature Machine (RFM) activation steering, which learns a targetdiscriminative direction from labeled forwardprocess activations. During sampling, noise alignment provides coarse control at high noise, while the RFM direction is reused over intermediate/late timesteps through lightweight activation edits. Experiments on CIFAR-10, ImageNet, CelebA, and fine-grained bird species show improved target accuracy over gradient-based posthoc guidance baselines, improved FID on the class-guidance benchmarks, and substantial inference speedups. Code: https://github. com/isotrivial/na-rfm.