Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering
Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, Wenzel Jakob
Abstract
Physically based differentiable rendering has recently evolved into a powerful tool for solving inverse problems involving light. Methods in this area perform a differentiable simulation of the physical process of light transport and scattering to estimate partial derivatives relating scene parameters to pixels in the rendered image. Together with gradient-based optimization, such algorithms have interesting applications in diverse disciplines, e.g., to improve the reconstruction of 3D scenes, while accounting for interreflection and transparency, or to design meta-materials with specified optical properties. The most versatile differentiable rendering algorithms rely on reverse-mode differentiation to compute all requested derivatives at once, enabling optimization of scene descriptions with millions of free parameters. However, a severe limitation of the reverse-mode approach is that it requires a detailed transcript of the computation that is subsequently replayed to back-propagate derivatives to the scene parameters. The transcript of typical renderings is extremely large, exceeding the available system memory by many orders of magnitude, hence current methods are limited to simple scenes rendered at low resolutions and sample counts. We introduce radiative backpropagation , a fundamentally different approach to differentiable rendering that does not require a transcript, greatly improving its scalability and efficiency. Our main insight is that reverse-mode propagation through a rendering algorithm can be interpreted as the solution of a continuous transport problem involving the partial derivative of radiance with respect to the optimization objective. This quantity is "emitted" by sensors, "scattered" by the scene, and eventually "received" by objects with differentiable parameters. Differentiable rendering then decomposes into two separate primal and adjoint simulation steps that scale to complex scenes rendered at high resolutions. We also investigated biased variants of this algorithm and find that they considerably improve both runtime and convergence speed. We showcase an efficient GPU implementation of radiative backpropagation and compare its performance and the quality of its gradients to prior work.
Ask Lune
Know where this paper stands in minutes, not an afternoon.
Lune can read the top-tier papers that cite it and the ones closest to it, and tells you what it claims, what came after and where it falls short, each point cited to the paper it comes from. No skimming, no AI slop.
34 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
its abstract · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising cites DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
- Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and Denoising cites Extracting Triangular 3D Models, Materials, and Lighting From Images
- Extracting Triangular 3D Models, Materials, and Lighting From Images cites DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
Cited by top-tier papers34
- Extracting Triangular 3D Models, Materials, and Lighting From ImagesJacob Munkberg, Wenzheng Chen, Jon Hasselgren, Alex Evans et al.CVPR 2022 · 306 citations
- Shape, Light, and Material Decomposition from Images using Monte Carlo Rendering and DenoisingJon Hasselgren, Nikolai Hofmann, Jacob MunkbergNeurIPS 2022 · 234 citations
- DR.JIT: a just-in-time compiler for differentiable renderingWenzel Jakob, Sébastien Speierer, Nicolas Roussel, Delio ViciniSIGGRAPH 2022 · 160 citations
- IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric ImagesKai Zhang, Fujun Luan, Zhengqi Li, Noah SnavelyCVPR 2022 · 85 citations
- DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable RendererWenzheng Chen, Joey Litalien, Jun Gao, Zian Wang et al.NeurIPS 2021 · 74 citations
Related papers
- Path replay backpropagation: differentiating light paths using constant memory and linear timeDelio Vicini, Sébastien Speierer, Wenzel JakobSIGGRAPH 2021 · 97 citations
- Monte Carlo estimators for differential light transportTizian Zeltner, Sébastien Speierer, Iliyan Georgiev, Wenzel JakobSIGGRAPH 2021 · 70 citations
- Differentiable Geometric Acoustic Path Tracing using Time-Resolved Path Replay BackpropagationUgo Paavo Finnendahl, Markus Worchel, Tobias Jüterbock, Daniel Wujecki et al.SIGGRAPH 2025 · 3 citations
- Radiance Caching for Differentiable Path TracingZiyi Zhang, Delio Vicini, Sebastian Winberg, Stephan J. Garbin et al.SIGGRAPH 2026
- Path-space differentiable rendering of participating mediaCheng Zhang, Zihan Yu, Shuang ZhaoSIGGRAPH 2021 · 50 citations
