EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

Download Video: HD (MP4, 3 MB)

Figure 2: As the baseline we use the approach of first recovering traditional RGB frames from events using E2VID and then learning regular NeRF with them as the training data. Note that such approach does not account for the sparsity and asynchronousity of the event stream. Hence, they need much more memory and disk space to store all the reconstructed views. Moreover, there is a limit to how short the E2VID window can be made, and hence to the number of reconstructed views. In contrast, EventNeRF respects the asynchronous nature of event streams and reconstructs the neural representation directly from the event stream. Moreover, it can use an arbitrary number of windows, which allows it to reconstruct the scene even from 3% of the data, as we show in Sec. 4.5. That makes it significantly more scalable than first reconstructing the frames and using traditional NeRF.

Download Videos: first (MP4, 3 MB) second (MP4, 3 MB)

Figure 3: We compare against the approach of first recovering traditional RGB frames from events using E2VID [36] and then learning regular NeRF with them as the training data. Our method clearly outperforms the E2VID+NeRF approach in all metrics and on all sequences and resembles the ground truth better.

Table 1: Comparing our method against E2VID+NeRF. Our method consistently produces better results.

Figure 4: Real data recording setup. The object is placed on a 45 RPM direct-drive vinyl turntable and lit by a ring light mounted directly above it. The scene is recorded with a DAVIS 346C colour event camera (right bottom).

Download Video: HD (MP4, 1 MB)

Figure 5: Showing the fact that our real sequences were recorded under low lighting conditions. Here, we show the RGB stream of the event camera that was used for recording the real sequences. These sequences were recorded using just a 5W USB light source. We tone map the real sequences in the remaining figures that use this RGB stream for clarity. Note that EventNeRF only uses the event stream and does not use the RGB stream.

Download Video: HD (MP4, 4 MB)

Figure 6: Real data results. In the “Plant” scene, we can reconstruct every stem and thin leaf. In the “Sewing” scene, we recover even a one-pixel-wide needle of the machine (best viewed with zoom). In the “Microphone” scene, we can reconstruct fine details such as the microphone grid. In the “Controller” scene, we preserve its details in the dark regions despite having low contrast. Similarly, in the “Goatling” scene, we can reconstruct both the details in the dark and bright highlights on the glasses. In the “Cube” scene, EventNeRF recovers sharp colour details. “Multimeter”, “Cube” and “Sewing” show how we recover view-dependent effects. In the “Bottle” scene, we can see the drawings on the reconstructed label. Note that all our results are halo-free, meaning they have no blurry silhouettes or trails behind the reconstructed object, which is a common artefact in event-based vision research.

Download Video: HD (MP4, 3 MB)

Figure 7: Results generated by different approaches. Our approach EventNeRF clearly outperforms all the methods. E2VID+NeRF produces significant artefacts in dark areas, uniformly-coloured surfaces, and also artefacts due to having a low number of reconstructed views used as NeRF training data. ssl-E2VID+NeRF does not support coloured data, handles short windows poorly, and has significant halos and trails behind the reconstructed object. In contrast, EventNeRF results are coloured, halo-free, detailed in both dark and bright regions, and the method works well with uniformly coloured surfaces.

Download Video: HD (MP4, 3 MB)

Figure 8: We also compare against Deblur-NeRF, i.e., a RGB-based NeRF extraction method designed specifically to handle blurry RGB videos. We perform this comparison by applying Deblur-NeRF on the the RGB stream of the event camera and synthetic RGB frames with blur simulated using the same exposure timestamps as in the real captures. Results, however, show that our approach significantly outperform Deblur-NeRF. This is expected, Deblur-NeRF can only handle view-inconsistent blur. On the other hand, our approach produces almost blur-free results. In addition, it is significantly more memory and computationally efficient. For Deblur-NeRF, 100 training views were used, which took around 22 seconds to record due to the low-light condition (see Fig. 5). Our EventNeRF approach, however, need only one 1.33s revolution of the object. In addition, it converges well within 6 hours using two NVIDIA Quadro RTX 8000 GPUs. This compares favourably to Deblur-NeRF, which needs 16 hours with the same GPU resources. Furthermore, the training time for our method significantly drops with the torch-ngp implementation to just 1 minute using a single NVIDIA GeForce RTX 2070 GPU (see Fig. 12 below).

Download Video: HD (MP4, 6 MB)

Figure 10: PSNR performance of our EventNeRF on “Drums” with varying event numbers (multiplied by 10⁶ on the x-axis). The images underneath the function graph show exemplary novel views obtained with different numbers of the events. To vary the number of events we only change the event generation threshold, the duration of training data remains constant.

Download Mesh: HD (GLB, 14 MB)

Figure 11: Using Marching Cubes, we can extract the learned geometry of "Chair" scene as a coloured mesh. Even though the original resolution of the training data is still only 346x260, the model and the extracted mesh are detailed.

Download Video: HD (MP4, 5 MB)

Figure 12: We show an interactive application of our approach that runs in real time. For this, we implement our method using torch-ngp instead of NeRF++ used for the main implementation. Training a model using this implementation takes around a minute using a single NVIDIA GeForce RTX 2070 GPU. At test, the method runs in real time. It produces highly photorealistic results that can be viewed from an arbitrary viewpoint. This, however, can come with some trade-off in the rendering quality.

Table 3: Comparing EventNeRF using the NeRF++ implementation against a real-time implementation based on torch-ngp. While the real-time implementation takes significantly less training and testing time, it can compromise on some of the rendering quality.

Download Video: HD (MP4, 49 MB)