Lune

SIGGRAPH2022Top-tier venue

DR.JIT: a just-in-time compiler for differentiable rendering

Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Delio Vicini

Abstract

derivative of a conceptually simple algorithm like path tracing [Kajiya 1986] with precautions for linear time complexity [Vicini et al. 2021] and unbiased visibility handling [Bangaru et al. 2020] turns into an enormously complicated function. Correct implementation of such a large and intricate program is near-impossible even for experts in the field. Mere correctness is also unsatisfactory: optimizations tend to run for thousands of iterations, hence the resulting program needs to be fast. It is evident that better tools are need bridge this conspicuous gap between PBDR theory and practice.

The design of Dr.Jit was guided by a single unifying objective: it should provide a practical and efficient foundation for work in this area. Most architectural decisions are direct consequences of this overarching goal. For example, consider the differentiation step that is implicit in differentiable rendering. Manual differentiation is tedious and error-prone, hence it is logical that the system should build on automatic differentiation (AD) to simplify development.

However, the needs of PBDR are more specific: standard use of AD to differentiate a rendering algorithm produces an inefficient and biased derivative that precludes many applications. Recent work addresses inefficiencies using physical reciprocity [Nimier-David et al. 2020] and arithmetic invertibility [Vicini et al. 2021] to turn the derivative of a simulation into a simulation of the derivative, while re-parameterizing the integration domain to remove bias [Loubet et al. 2019]. These steps move the differentiation operation into the random walk, where it introduces partial derivative terms at each scattering event. This has implications on the design of the system: the derivatives must somehow be (pre-)compiled, since the machinery of AD is too slow to be used dynamically at such high rates.

Differentiating a simulation changes the underlying computation, but the details of this change depend on the scene, simulation algorithm, and optimization task. When the optimization only targets a subset of the scene's parameters, it is desirable that the system uses this information to remove steps that cannot influence the computed gradient. The dynamic nature of this problem calls for a similarly dynamic approach to compilation, which is why we pursue an approach centered around just-in-time (JIT) compilation.

Effective use of modern computing hardware requires that the program is organized into data-parallel phases known as kernels. Several kernels are generally needed to handle data dependencies, which must then exchange information through device memory. This interkernel communication comes at a cost in terms of storage and memory bandwidth, hence the specific manner in which a computation is partitioned into kernels can have a pronounced impact on performance. In the case of PBDR, the simulation parallelizes over millions of Monte Carlo samples that represent a large amount of program state. In our experiments, we find that it is almost always preferable that the Monte Carlo integration occurs within a megakernel, i.e., a large kernel containing all program instructions needed to evaluate the integrand. Most sample state can then be stored in registers, reducing memory usage and inter-kernel communication.

Finally, physically-based rendering algorithms are commonly expressed using subtype polymorphism to dynamically dispatch method calls from abstract component interfaces (e.g., a material encountered by a ray) to concrete implementations (e.g., a woven fabric or a rough metallic surface). The ability to represent, differentiate, and optimize such polymorphic constructions benefits performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

14 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Conditional Mixture Path Guiding for Differentiable Rendering cites Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering
  • Conditional Mixture Path Guiding for Differentiable Rendering cites Recursive Control Variates for Inverse Rendering
  • Conditional Mixture Path Guiding for Differentiable Rendering cites Path replay backpropagation: differentiating light paths using constant memory and linear time
  • Conditional Mixture Path Guiding for Differentiable Rendering cites Path-space differentiable rendering
  • Recursive Control Variates for Inverse Rendering cites Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering
  • Recursive Control Variates for Inverse Rendering cites Path replay backpropagation: differentiating light paths using constant memory and linear time
  • Recursive Control Variates for Inverse Rendering cites Path-space differentiable rendering
  • NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Rendering cites Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering
  • NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Rendering cites Recursive Control Variates for Inverse Rendering
  • NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Rendering cites Path replay backpropagation: differentiating light paths using constant memory and linear time
  • NeRF as a Non-Distant Environment Emitter in Physics-based Inverse Rendering cites Path-space differentiable rendering
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers14

  1. Recursive Control Variates for Inverse RenderingBaptiste Nicolet, Fabrice Rousselle, Jan Novák, Alexander Keller et al.SIGGRAPH 2023 · 32 citations
  2. NeuSample: Importance Sampling for Neural MaterialsBing Xu, Liwen Wu, Milos Hasan, Fujun Luan et al.SIGGRAPH 2023 · 19 citations
  3. NeRF as a Non-Distant Environment Emitter in Physics-based Inverse RenderingJingwang Ling, Ruihan Yu, Feng Xu, Chun Du et al.SIGGRAPH 2024 · 14 citations
  4. N-Dimensional Gaussians for Fitting of High Dimensional FunctionsStavros Diolatzis, Tobias Zirr, Alexander Kuznetsov, Georgios Kopanas et al.SIGGRAPH 2024 · 8 citations
  5. Conditional Mixture Path Guiding for Differentiable RenderingZhimin Fan, Pengcheng Shi, Mufan Guo, Ruoyu Fu et al.SIGGRAPH 2024 · 7 citations

Builds on7

  1. DiffTaichi: Differentiable Programming for Physical SimulationYuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun et al.ICLR 2020 · 479 citations
  2. Path-space differentiable renderingCheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas et al.SIGGRAPH 2020 · 155 citations
  3. Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast GradientsWilliam S. Moses, Valentin ChuravyNeurIPS 2020 · 144 citations
  4. Radiative backpropagation: an adjoint method for lightning-fast differentiable renderingMerlin Nimier-David, Sébastien Speierer, Benoît Ruiz, Wenzel JakobSIGGRAPH 2020 · 107 citations
  5. Path replay backpropagation: differentiating light paths using constant memory and linear timeDelio Vicini, Sébastien Speierer, Wenzel JakobSIGGRAPH 2021 · 97 citations

Related papers

  1. Monte Carlo estimators for differential light transportTizian Zeltner, Sébastien Speierer, Iliyan Georgiev, Wenzel JakobSIGGRAPH 2021 · 70 citations
  2. Systematically differentiating parametric discontinuitiesSai Praveen Bangaru, Jesse Michel, Kevin Mu, Gilbert Bernstein et al.SIGGRAPH 2021 · 30 citations
  3. Plateau-Reduced Differentiable Path TracingMichael Fischer, Tobias RitschelCVPR 2023
  4. Path-Space Differentiable Rendering of Implicit SurfacesSiwei Zhou, Youngha Chang, Nobuhiko Mukai, Hiroaki Santo et al.SIGGRAPH 2024 · 1 citation
  5. Distributions for Compositionally Differentiating Parametric DiscontinuitiesJesse Michel, Kevin Mu, Xuanda Yang, Sai Praveen Bangaru et al.OOPSLA 2024 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines