Lune

NeurIPS2025Top-tier venue

Collapsing Taylor Mode Automatic Differentiation

Felix Dangel, Tim Siebert, Marius Zeinhofer, Andrea Walther

Abstract

Computing partial differential equation (PDE) operators via nested backpropagation is expensive, yet popular, and severely restricts their utility for scientific machine learning. Recent advances, like the forward Laplacian and randomizing Taylor mode automatic differentiation (AD), propose forward schemes to address this. We introduce an optimization technique for Taylor mode that'collapses'derivatives by rewriting the computational graph, and demonstrate how to apply it to general linear PDE operators, and randomized Taylor mode. The modifications simply require propagating a sum up the computational graph, which could -- or should -- be done by a machine learning compiler, without exposing complexity to users. We implement our collapsing procedure and evaluate it on popular PDE operators, confirming it accelerates Taylor mode and outperforms nested backpropagation.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Unbiased and Second-Order-Free Training for High-Dimensional PDEs cites Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operators
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Unbiased and Second-Order-Free Training for High-Dimensional PDEsJaemin Seo, Su Rin Lee, JaeYong LeeICML 2026 · 1 citation

Builds on2

  1. Stochastic Taylor Derivative Estimator: Efficient amortization for arbitrary differential operatorsZekun Shi, Zheyuan Hu, Min Lin, Kenji KawaguchiNeurIPS 2024 · 32 citations
  2. Kronecker-Factored Approximate Curvature for Physics-Informed Neural NetworksFelix Dangel, Johannes Müller, Marius ZeinhoferNeurIPS 2024 · 31 citations

Related papers

  1. Randomized Automatic DifferentiationDeniz Oktay, Nick McGreivy, Joshua Aduol, Alex Beatson et al.ICLR 2021 · 31 citations
  2. AD for an Array Language with Nested ParallelismRobert Schenck, Ola Rønning, Troels Henriksen, Cosmin E. OanceaSC 2022 · 11 citations
  3. Gradients of Functions of Large MatricesNicholas Krämer, Pablo Moreno-Muñoz, Hrittik Roy, Søren HaubergNeurIPS 2024 · 6 citations
  4. Efficient Learning of PDEs via Taylor Expansion and Sparse Decomposition into Value and Fourier DomainsMd. Nasim, Yexiang XueAAAI 2024
  5. ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct IndexingShuhong Huang, Shizhi Tang, Yuan Wen, Huanqi Cao et al.PPoPP 2026
Dusk over the sea between two cliffs drawn in fine vertical lines