Lune

SIGGRAPH2023Top-tier venue

Differentiable Heightfield Path Tracing with Accelerated Discontinuities

Xiaochun Tong, Hsueh-Ti Derek Liu, Yotam I. Gingold, Alec Jacobson

Abstract

We investigate the problem of accelerating a physically-based differentiable renderer for heightfields based on path tracing with global illumination. On a heightfield with 1 million vertices (1024 × 1024 resolution), our differentiable renderer requires only 4 ms per sample per pixel when differentiating direct illumination, orders of magnitude faster than most existing general 3D mesh differentiable renderers. It is well-known that one can leverage spatial hierarchical data structures (e.g., the maximum mipmaps) to accelerate the forward pass of heightfield rendering. The key idea of our approach is to further utilize the hierarchy to speed up the backward pass—differentiable heightfield rendering. Specifically, we use the maximum mipmaps to accelerate the process of identifying scene discontinuities, which is crucial for obtaining accurate derivatives. Our renderer supports global illumination. we are able to optimize global effects, such as shadows, with respect to the geometry and the material parameters. Our differentiable renderer achieves real-time frame rates and unlocks interactive inverse rendering applications. We demonstrate the flexibility of our method with terrain optimization, geometric illusions, shadow optimization, and text-based shape generation.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Learning Deformable Tetrahedral Meshes for 3D Reconstruction cites Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning
  • DR.JIT: a just-in-time compiler for differentiable rendering cites Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Practical Stylized Nonlinear Monte Carlo RenderingXiaochun Tong, Toshiya HachisukaSIGGRAPH 2025 · 2 citations

Builds on12

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Soft Rasterizer: A Differentiable Renderer for Image-Based 3D ReasoningShichen Liu, Weikai Chen, Tianye Li, Hao LiICCV 2019 · 789 citations
  3. DR.JIT: a just-in-time compiler for differentiable renderingWenzel Jakob, Sébastien Speierer, Nicolas Roussel, Delio ViciniSIGGRAPH 2022 · 160 citations
  4. Learning Deformable Tetrahedral Meshes for 3D ReconstructionJun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson et al.NeurIPS 2020 · 134 citations
  5. Radiative backpropagation: an adjoint method for lightning-fast differentiable renderingMerlin Nimier-David, Sébastien Speierer, Benoît Ruiz, Wenzel JakobSIGGRAPH 2020 · 107 citations

Related papers

  1. Monte Carlo estimators for differential light transportTizian Zeltner, Sébastien Speierer, Iliyan Georgiev, Wenzel JakobSIGGRAPH 2021 · 70 citations
  2. Antithetic sampling for Monte Carlo differentiable renderingCheng Zhang, Zhao Dong, Michael C. Doggett, Shuang ZhaoSIGGRAPH 2021 · 55 citations
  3. Path-space differentiable renderingCheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas et al.SIGGRAPH 2020 · 155 citations
  4. Differentiable Shadow Mapping for Efficient Inverse GraphicsMarkus Worchel, Marc AlexaCVPR 2023
  5. Quadric-Based Silhouette Sampling for Differentiable RenderingMariia Soroka, Christoph Peters, Steve MarschnerSIGGRAPH 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines