Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients
Abstract
Applying differentiable programming techniques and machine learning algorithms to foreign programs requires developers to either rewrite their code in a machine learning framework, or otherwise provide derivatives of the foreign code. This paper presents Enzyme 1 , a high-performance automatic differentiation (AD) compiler plugin for the LLVM compiler framework capable of synthesizing gradients of statically analyzable programs expressed in the LLVM intermediate representation (IR). Enzyme synthesizes gradients for programs written in any language whose compiler targets LLVM IR including C, C++, Fortran, Julia, Rust, Swift, MLIR, etc., thereby providing native AD capabilities in these languages. Unlike traditional source-to-source and operator-overloading tools, Enzyme performs AD on optimized IR. On a machine-learning focused benchmark suite including Microsoft's ADBench, AD on optimized IR achieves a geometric mean speedup of 4.5x over AD on IR before optimization allowing Enzyme to achieve state-of-the-art performance. Packaging Enzyme for PyTorch and TensorFlow provides convenient access to gradients of foreign code with state-of-the art performance, enabling foreign code to be directly incorporated into existing machine learning workflows.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- AD for an Array Language with Nested Parallelism cites Reverse-mode automatic differentiation and optimization of GPU kernels via enzyme
- DR.JIT: a just-in-time compiler for differentiable rendering cites Reverse-mode automatic differentiation and optimization of GPU kernels via enzyme
- Scalable Automatic Differentiation of Multiple Parallel Paradigms through Compiler Augmentation cites AD for an Array Language with Nested Parallelism
- Scalable Automatic Differentiation of Multiple Parallel Paradigms through Compiler Augmentation cites Reverse-mode automatic differentiation and optimization of GPU kernels via enzyme
Cited by top-tier papers8
- DR.JIT: a just-in-time compiler for differentiable renderingWenzel Jakob, Sébastien Speierer, Nicolas Roussel, Delio ViciniSIGGRAPH 2022 · 160 citations
- Reverse-mode automatic differentiation and optimization of GPU kernels via enzymeWilliam S. Moses, Valentin Churavy, Ludger Paehler, Jan Hückelheim et al.SC 2021 · 50 citations
- Scalable Automatic Differentiation of Multiple Parallel Paradigms through Compiler AugmentationWilliam S. Moses, Sri Hari Krishna Narayanan, Ludger Paehler, Valentin Churavy et al.SC 2022 · 25 citations
- AD for an Array Language with Nested ParallelismRobert Schenck, Ola Rønning, Troels Henriksen, Cosmin E. OanceaSC 2022 · 11 citations
- Closed-Form Diffeomorphic Transformations for Time Series AlignmentIñigo Martinez, Elisabeth Viles, Igor G. OlaizolaICML 2022 · 11 citations
Related papers
- Fuzzing Automatic Differentiation in Deep-Learning LibrariesChenyuan Yang, Yinlin Deng, Jiayi Yao, Yuxing Tu et al.ICSE 2023 · 34 citations
- Sound and Modular Activity Analysis for Automatic Differentiation in MLIRMai Jacob Peng, William S. Moses, Oleksandr Zinenko, Christophe DubachOOPSLA 2025
- Aδ: autodiff for discontinuous programs - applied to shadersYuting Yang, Connelly Barnes, Andrew Adams, Adam FinkelsteinSIGGRAPH 2022 · 17 citations
- 𝜆ₛ: computable semantics for differentiable programming with higher-order functions and datatypesBenjamin Sherman, Jesse Michel, Michael CarbinPOPL 2021 · 11 citations
- MAGPY: Compiling Eager Mode DNN Programs by Monitoring Execution StatesChen Zhang, Rongchao Dong, Haojie Wang, Runxin Zhong et al.USENIX ATC 2024 · 5 citations
