Collapsing Taylor Mode Automatic Differentiation
Abstract
Computing partial differential equation (PDE) operators via nested backpropagation is expensive, yet popular, and severely restricts their utility for scientific machine learning. Recent advances, like the forward Laplacian and randomizing Taylor mode automatic differentiation (AD), propose forward schemes to address this. We introduce an optimization technique for Taylor mode that'collapses'derivatives by rewriting the computational graph, and demonstrate how to apply it to general linear PDE operators, and randomized Taylor mode. The modifications simply require propagating a sum up the computational graph, which could -- or should -- be done by a machine learning compiler, without exposing complexity to users. We implement our collapsing procedure and evaluate it on popular PDE operators, confirming it accelerates Taylor mode and outperforms nested backpropagation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on2
- Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operatorsZekun Shi, Zheyuan Hu, Min Lin, Kenji KawaguchiNeurIPS 2024 · 32 citations
- Kronecker-Factored Approximate Curvature for Physics-Informed Neural NetworksFelix Dangel, Johannes Müller, Marius ZeinhoferNeurIPS 2024 · 31 citations
Related papers
- Randomized Automatic DifferentiationDeniz Oktay, Nick McGreivy, Joshua Aduol, Alex Beatson et al.ICLR 2021 · 31 citations
- AD for an Array Language with Nested ParallelismRobert Schenck, Ola Rønning, Troels Henriksen, Cosmin E. OanceaSC 2022 · 11 citations
- Gradients of Functions of Large MatricesNicholas Krämer, Pablo Moreno-Muñoz, Hrittik Roy, Søren HaubergNeurIPS 2024 · 6 citations
- Efficient Learning of PDEs via Taylor Expansion and Sparse Decomposition into Value and Fourier DomainsMd. Nasim, Yexiang XueAAAI 2024
- ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct IndexingShuhong Huang, Shizhi Tang, Yuan Wen, Huanqi Cao et al.PPoPP 2026
