Neural Radiance Fields for Outdoor Scene Relighting

Figure 2: Our NeRF-OSR uses outdoor images of a site photographed in an uncontrolled setting (dashed green) to recover a relightable implicit scene model. It learns the scene intrinsics and illumination as expressed by the spherical harmonics (SH) coefficients. Here, a dedicated neural component learns shadows. During the test, our technique can synthesise novel images at arbitrary camera viewpoints and scene illumination; the user directly supplies the desired camera pose and the scene illumination, either from an environment map or directly via SH coefficients.

Figure 3: Sample views from the new benchmark dataset for outdoor scene relighting. Our dataset contains eight sites shot from various viewpoints and at different timings using a DSLR camera and a 360° camera to capture the environment map. A colour chequerboard is also captured from both the DSLR and 360° cameras to account for colour calibration (see Figure 4). The dataset has 3240 views captured in 110 different recording sessions. The dataset is publicly available in the Downloads section below.

Figure 4: For each recording session in our new benchmark dataset, we capture a colour chequerboard from the DSLR (a) and from the 360° (b) cameras. The chequerboards are used to colour-correct the environment maps with respect to the DSLR (see (e) and (f) for before and after correction). This allows for the first time accurate numerical evaluations of outdoor scene relighting methods on real data against ground truth. Note that (c) shows the original colour chequerboard of (b) being reprojected from the 360° view into the regular view, and its resulting colour-corrected version is shown in (d).

Figure 5a: Site 1 (Ludwigskirche, Saarbrücken).

Figure 5b: Site 2 (Staatstheater, Saarbrücken).

Figure 5c: Site 3 (Landwehrplatz, Saarbrücken).

Figure 5d: Trevi Fountain (Phototourism dataset).

Figure 6: We compare our technique against the work of Yu et al. Not only we can produce results of much higher resolution than their method, but they also look significantly more natural.

Figure 7: NeRF-W can only interpolate between different learned appearances. Moreover, it does not allow for semantic control over the appearance. In contrast, our technique decomposes the scene into multiple intrinsics, including albedo, normals, shadows, and illumination. This allows for direct manipulation of the lighting conditions. Moreover, we show that our technique generalises to the conditions impossible in real life by rotating the estimated lighting parameters in the full 360 degrees circle around the fountain.

Table 1: Quantitative evaluation of the relighting capabilities of different techniques. Here, we report the mean reconstruction error on Site 1 from our dataset. Our technique significantly outperforms related methods (Yu et al., Philip et al.). Bottom: ablation study of our various design choices. Our full model achieves the best result.

Figure 8: The proposed approach generalises to highly unnatural illuminations as well. This can be leveraged for creation of various visual effects.

Figure 9: Another application of our intrinsic decomposition is to edit the scene albedo, without affecting the illumination or shadows. Such application is not possible by style-based methods by any means, e.g., NeRF-W, as they do not perform image decomposition. In this video, we replace the announcement poster in Site 3 with an ECCV 2022 poster. Note how the replaced poster looks natural with the rest of the scene.

Figure 10: Another application of our intrinsic decomposition is to edit the shadow strength post-render, without affecting the illumination or albedo. Such application is not possible by style-based methods by any means, e.g., NeRF-W, as they do not perform image decomposition.

Figure 11: In contrast to style-based methods such as NeRF-W, our rendering is an explicit function of geometry, albedo, shadow, and the lighting conditions (see Eq. 5). Our model provides direct access to albedo and geometry. The lighting and shadows can be generated from the geometry using multiple, potentially non-differentiable, lighting models at render-time. Hence, if we can extract geometry and albedo at sufficient resolution, we can use them without the slow NeRF ray-marching at little to no loss of quality, compared to the original neural model. We extract the geometry and albedo from the learned model of Site 1 as a mesh using Marching Cubes at resolution 1000³ voxels. Then we use them in our interactive VR renderer implemented with C++, OpenGL and SteamVR.

Download Video: HD (MP4, 64 MB)