MOD-SR: Unifying Multimodal Learning and Direct Optimization with Gradient-Guided Diffusion Model for Symbolic Regression

Symbolic regression (SR) aims to discover interpretable mathematical expressions from observed data. While recent generative approaches have shown promise in treating SR as machine translation or multimodal learning tasks using NN methods, they suffer from uncontrollable generation process and training-evaluation misalignment. The training objectives (average cross-entropy loss on a token level across the distribution of historical data) differ from the evaluation metric (fitting error for every test data / complexity), necessitating extensive heuristic post-processing. On the other hand, direct optimization methods suffer from an exponential slowdown as the dimensionality increases, non-differentiability and local optima traps. We propose MOD-SR, unifying multimodal distribution learning during training with direct optimization at inference time. This is achieved by modeling the task as $p(x_0 \mid \mathcal{D}, y^*)$ and employing gradient-guided diffusion in embedding space, enhanced by contrastive learning and representation alignment. Furthermore, we introduce DFEX, a fixed-depth tree relaxation method that ensures differentiability for effective gradient guidance during inference. Experiments demonstrate that MOD-SR achieves superior performance on diverse benchmarks through a unified framework integrating distribution learning and optimization. Our code is available at https://github.com/KROX777/MOD-SR.