Figure 1: We present the first neural radiance fields for outdoor scene relighting. From uncontrolled outdoor images, we learn a neural representation of scene geometry, albedo and illumination-dependent shadows from which we can edit both lighting and viewpoint.

Abstract

Photorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination. A delicate manipulation of the lighting can then be performed while keeping the scene albedo and geometry unaltered. We present NeRF-OSR, i.e., the first approach for outdoor scene relighting based on neural radiance fields. In contrast to the prior art, our technique allows simultaneous editing of both scene illumination and camera viewpoint using only a collection of outdoor photos shot in uncontrolled settings. Moreover, it enables direct control over the scene illumination, as defined through a spherical harmonics model. It also includes a dedicated network for shadow reproduction, which is crucial for high-quality outdoor scene relighting. To evaluate the proposed method, we collect a new benchmark dataset of several outdoor sites, where each site is photographed from multiple viewpoints and at different timings. For each timing, a 360° environment map is captured together with a colour-calibration chequerboard to allow accurate numerical evaluations on real data against ground truth. Comparisons against state of the art show that NeRF-OSR enables controllable light and viewpoint editing at higher quality and with realistic self-shadowing reproduction. Our method and the dataset will be made publicly available.

Method

Figure 2: Our NeRF-OSR uses outdoor images of a site photographed in an uncontrolled setting (dashed green) to recover a relightable implicit scene model. It learns the scene intrinsics and illumination as expressed by the spherical harmonics (SH) coefficients. Here, a dedicated neural component learns shadows. During the test, our technique can synthesise novel images at arbitrary camera viewpoints and scene illumination; the user directly supplies the desired camera pose and the scene illumination, either from an environment map or directly via SH coefficients.

Dataset

Figure 3: Sample views from the new benchmark dataset for outdoor scene relighting. Our dataset contains eight sites shot from various viewpoints and at different timings using a DSLR camera and a 360° camera to capture the environment map. A colour chequerboard is also captured from both the DSLR and 360° cameras to account for colour calibration (see Figure 4). The dataset has 3240 views captured in 110 different recording sessions. The dataset will be made publicly available.

Figure 4: For each recording session in our new benchmark dataset, we capture a colour chequerboard from the DSLR (a) and from the 360° (b) cameras. The chequerboards are used to colour-correct the environment maps with respect to the DSLR (see (e) and (f) for before and after correction). This allows for the first time accurate numerical evaluations of outdoor scene relighting methods on real data against ground truth. Note that (c) shows the original colour chequerboard of (b) being reprojected from the 360° view into the regular view, and its resulting colour-corrected version is shown in (d).

Novel Lighting and Viewpoint

Figure 5a: Site 1 (Ludwigskirche, Saarbrücken).

Figure 5b: Site 2 (Staatstheater, Saarbrücken).

Figure 5c: Site 3 (Landwehrplatz, Saarbrücken).

Figure 5d: Trevi Fountain (Phototourism dataset).

Unnatural Illumination

Figure 6: The proposed approach generalises to highly unnatural illuminations as well. This can be leveraged for creation of various visual effects.

Qualitative Comparison with Yu et al.

Figure 7: We compare our technique against the work of Yu et al. Not only we can produce results of much higher resolution than their method, but they also look significantly more natural.

Qualitative Comparison with NeRF-W (Martin-Brualla, Radwan, Sajjadi et al.)

Figure 8: NeRF-W can only interpolate between different learned appearances. Moreover, it does not allow for semantic control over the appearance. In contrast, our technique decomposes the scene into multiple intrinsics, including albedo, normals, shadows, and illumination. This allows for direct manipulation of the lighting conditions. Moreover, we show that our technique generalises to the conditions impossible in real life by rotating the estimated lighting parameters in the full 360 degrees circle around the fountain.

Quantitative Comparison with Yu et al. and Philip et al.

Method PSNR↑ MSE↓ MAP↓
Yu et al. 18.71 0.0138 0.0881
Philip et al. (downscaled) 17.37 0.0194 0.1046
Ours (downscaled) 19.86 0.0114 0.0802
Yu et al. (upscaled) 17.87 0.0167 0.0967
Philip et al. 16.63 0.0229 0.1131
Ours 18.72 0.0143 0.0893
No shadows 17.82 0.0172 0.1012
No annealing 17.16 0.0195 0.1082
No ray jitter 18.43 0.0150 0.0931
No shadow jitter 18.28 0.0155 0.0954
No shadow regulariser 17.62 0.0181 0.1046

Table 1: Quantitative evaluation of the relighting capabilities of different techniques. Here, we report the mean reconstruction error on Site 1 from our dataset. Our technique significantly outperforms related methods (Yu et al., Philip et al.). Bottom: ablation study of our various design choices. Our full model achieves the best result.

Narrated Video with Experiments

Download Video: HD (MP4, 33 MB)

Downloads


Citation

BibTeX, 1 KB

@misc{rudnev2021nerfosr,
      title={Neural Radiance Fields for Outdoor Scene Relighting},
      author={Viktor Rudnev and Mohamed Elgharib and William Smith and Lingjie Liu and Vladislav Golyanik and Christian Theobalt},
      year={2021},
      eprint={2112.05140},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Contact

For questions, clarifications, please get in touch with:
Viktor Rudnev
vrudnev@mpi-inf.mpg.de
Vladislav Golyanik
golyanik@mpi-inf.mpg.de
Mohamed Elgharib
elgharib@mpi-inf.mpg.de

This page is Zotero translator friendly. Imprint. Data Protection.