Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models
Congcong Zhu, Xiaoyan Xu, Jiayue Han, Jingrun Chen
Abstract
Auto-regressive partial differential equation (PDE) foundation models have shown great potential in handling time-dependent data. However, these models suffer from error accumulation caused by the shortcut problem deeply rooted in auto-regressive prediction. The challenge becomes particularly evident for out-of-distribution data, as the pretraining performance may approach random model initialization for downstream tasks with long-term dynamics. To deal with this problem, we propose physics-informed temporal alignment (PITA), a self-supervised learning framework inspired by inverse problem solving. Specifically, PITA aligns the physical dynamics discovered at different time steps on each given PDE trajectory by integrating physicsinformed constraints into the self-supervision signal. The alignment is derived from observation data without relying on known physics priors, indicating strong generalization ability to out-of-distribution data. Extensive experiments show that PITA significantly enhances the accuracy and robustness of existing foundation models on diverse time-dependent PDE data. The code is available at https://github.com/ SCAILab-USTC/PITA.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer Behavior cites Fourier Neural Operator for Parametric Partial Differential Equations
- ComPhy: Composing Physical Models with end-to-end Alignment cites Fourier Neural Operator for Parametric Partial Differential Equations
Cited by top-tier papers2
Builds on12
- Fourier Neural Operator for Parametric Partial Differential EquationsZongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu et al.ICLR 2021 · 3,911 citations
- Model Sparsity Can Simplify Machine UnlearningJinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao et al.NeurIPS 2023 · 293 citations
- Can contrastive learning avoid shortcut solutions?Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich et al.NeurIPS 2021 · 185 citations
- Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer BehaviorShashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji et al.NeurIPS 2023 · 173 citations
- Rectifying the Shortcut Learning of Background for Few-Shot LearningXu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang et al.NeurIPS 2021 · 110 citations
Related papers
- OmniArch: Building Foundation Model for Scientific ComputingTianyu Chen, Haoyi Zhou, Ying Li, Hao Wang et al.ICML 2025
- Poseidon: Efficient Foundation Models for PDEsMaximilian Herde, Bogdan Raonic, Tobias Rohner, Roger Käppeli et al.NeurIPS 2024 · 235 citations
- PINFDiT: Energy-Based Physics-Informed Diffusion Transformers for General-purpose Time Series TasksDefu Cao, Wen Ye, Yizhou Zhang, Sam Griesemer et al.ICLR 2026
- DPM: A Novel Training Method for Physics-Informed Neural Networks in ExtrapolationJungeun Kim, Kookjin Lee, Dongeun Lee, Sheo Yon Jhin et al.AAAI 2021 · 112 citations
- DISCO: learning to DISCover an evolution Operator for multi-physics-agnostic predictionRudy Morel, Jiequn Han, Edouard OyallonICML 2025
