Lune

AAAI2022Top-tier venue

Anisotropic Fourier Features for Neural Image-Based Rendering and Relighting

Huangjie Yu, Anpei Chen, Xin Chen, Lan Xu, Ziyu Shao, Jingyi Yu

Abstract

Recent neural rendering techniques have greatly benefited image-based modeling and relighting tasks. They provide a continuous, compact, and parallelable representation by modeling the plenoptic function as multilayer perceptrons (MLPs). However, vanilla MLPs suffer from spectral biases on multidimensional datasets. Recent rescues based on isotropic Fourier features mapping mitigate the problem but still fall short of handling heterogeneity across different dimensions, causing imbalanced regression and visual artifacts such as excessive blurs. We present an anisotropic random Fourier features (RFF) mapping scheme to tackle spectral biases. We first analyze the influence of bandwidth from a different perspective: we show that the optimal bandwidth exhibits strong correlations with the frequency spectrum of the training data across various dimensions. We then introduce an anisotropic feature mapping scheme with multiple bandwidths to model the multidimensional signal characteristics. We further propose an efficient bandwidth searching scheme through iterative golden-section search that can significantly reduce the training overload from polynomial time to logarithm. Our anisotropic scheme directly applies to neural surface light-field rendering and image-based relighting. Comprehensive experiments show that our scheme can more faithfully model lighting conditions and object features as well as preserve fine texture details and smooth view transitions even when angular and spatial samples are highly imbalanced.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains cites Reconstructing continuous distributions of 3D protein structure from cryo-EM images
  • Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains cites Frequency Bias in Neural Networks for Input of Non-Uniform Density
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Depth-Guided Robust and Fast Point Cloud Fusion NeRF for Sparse Input ViewsShuai Guo, Qiuwen Wang, Yijie Gao, Rong Xie et al.AAAI 2024 · 10 citations
  2. AltNeRF: Learning Robust Neural Radiance Field via Alternating Depth-Pose OptimizationKun Wang, Zhiqiang Yan, Huang Tian, Zhenyu Zhang et al.AAAI 2024 · 6 citations

Builds on9

  1. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional DomainsMatthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil et al.NeurIPS 2020 · 4,036 citations
  2. Implicit Neural Representations with Periodic Activation FunctionsVincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell et al.NeurIPS 2020 · 4,008 citations
  3. Multiview Neural Surface Reconstruction by Disentangling Geometry and AppearanceLior Yariv, Yoni Kasten, Dror Moran, Meirav Galun et al.NeurIPS 2020 · 1,010 citations
  4. Frequency Bias in Neural Networks for Input of Non-Uniform DensityRonen Basri, Meirav Galun, Amnon Geifman, David W. Jacobs et al.ICML 2020 · 229 citations
  5. Reconstructing continuous distributions of 3D protein structure from cryo-EM imagesEllen D. Zhong, Tristan Bepler, Joseph H. Davis, Bonnie BergerICLR 2020 · 124 citations

Related papers

  1. NeuRBF: A Neural Fields Representation with Adaptive Radial Basis FunctionsZhang Chen, Zhong Li, Liangchen Song, Lele Chen et al.ICCV 2023 · 80 citations
  2. Improved Implicit Neural Representation with Fourier Reparameterized TrainingKexuan Shi, Xingyu Zhou, Shuhang GuCVPR 2024 · 14 citations
  3. PlenOctrees for Real-time Rendering of Neural Radiance FieldsAlex Yu, Ruilong Li, Matthew Tancik, Hao Li et al.ICCV 2021 · 1,284 citations
  4. Deep Fourier-based Arbitrary-scale Super-resolution for Real-time RenderingHaonan Zhang, Jie Guo, Jiawei Zhang, Haoyu Qin et al.SIGGRAPH 2024 · 9 citations
  5. FINER: Flexible Spectral-Bias Tuning in Implicit NEural Representation by Variableperiodic Activation FunctionsZhen Liu, Hao Zhu, Qi Zhang, Jingde Fu et al.CVPR 2024
Dusk over the sea between two cliffs drawn in fine vertical lines