Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment

Despite their strong general capabilities, large language models (LLMs) often remain unreliable when outputs must be numerically precise. A key reason is the training objective: standard crossentropy treats numeric tokens as unstructured categories and ignores the metric structure of their values. We address this mismatch with Smooth Maximum Mean Discrepancy (SMMD), which builds on the classic MMD by incorporating valuedistance kernels over numeric tokens and graphbased smoothness. With this kernel defined over a numeric sub-vocabulary, SMMD aligns the predicted numeric distribution to the target via kernel matching and smooths the prediction-target residual over the induced kernel graph to encourage local consistency. We evaluate SMMD on four numeric-target tasks-mathematical reasoning, arithmetic calculation, clock-time recognition, and chart question answering-across multiple open-weight LLM and VLM backbones. SMMD consistently improves accuracy over both crossentropy and recent numeric-target losses; analyses show complementary effects between MMD and smoothness and underscore the importance of distance-based kernel design. Code is available at https://github.com/Zuozhuo/ smmd-loss .