When Compilation Breaks Your GNN: A Numerical Stability Perspective

Deep learning compilers like torch.compile accelerate GNNs through operator fusion and computation reordering, yet often introduce numerical instability that causes models to diverge under hyperparameters that work in eager mode. We trace this fragility to the interaction between compiler optimizations and GNN structure. Compilers reorder floating-point additions assuming associativity, but rounding errors from different summation orders, though typically negligible, can be catastrophically amplified by ill-conditioned computations. We show that GNN neighbor aggregation is structurally prone to such ill-conditioning: power-law degree distributions create highly variable aggregation lengths, while normalized features induce message cancellation during summation. This yields a long-tailed distribution of condition numbers across nodes, where a critical minority of aggregations amplify rounding differences into macroscopic output deviations. Based on this analysis, we propose Numerics-Aware Graph Compilation (NAGC), which estimates per-aggregation numerical risk through lightweight profiling and selectively applies aggressive optimizations only to well-conditioned operations. Experiments on Open Graph Benchmark datasets show that NAGC preserves over 95% of compilation speedups while reducing numerical deviation by 10 to 100 times and eliminating compilation-induced training failures.