4DQV

About us

Welcome to the web page of the 4D and Quantum Vision (4DQV) group led by Dr. Vladislav Golyanik.

The focus of our team lies on 3D reconstruction and analysis of general deformable scenes, 3D reconstruction of the human body and matching problems on point sets and graphs. We are interested in neural approaches (both supervised and unsupervised), physics-based methods as well as new hardware and sensors (e.g., quantum computers and event cameras). We are integrated in the Visual Computing and Artificial Intelligence Department (D6) and working closely with Prof. Christian Theobalt.

Many research questions at the intersection of computer graphics, computer vision and machine learning involve challenging search problems (e.g., graph matching) or the optimisation of non-convex objectives. For such problems, we develop new algorithmic formulations that can be solved on modern adiabatic quantum annealers or universal quantum computers and investigate which advantages these approaches offer compared to existing classical methods.

Our reserach interests include (but are not limited to):

3D Reconstruction and Tracking of Rigid and Non-Rigid Scenes and Objects
Neural Rendering
Point Set Registration and Matching Problems
Quantum Algorithms for Computer Vision and Graphics
Event-based Approaches in Vision and Graphics

Recent Projects

EgoLocate (SIGGRAPH 2023)

EgoLocate: Real-time Motion Capture, Localization, and Mapping with Sparse Body-mounted Sensors

QMMF (CVPR 2023)

Quantum Multi-Model Fitting

MoFusion (CVPR 2023)

MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis

EventNeRF (CVPR 2023)

EventNeRF: Neural Radiance Fields from a Single Colour Event Camera

CCuantuMM (CVPR 2023)

CCuantuMM: Cycle-Consistent Quantum-Hybrid Matching of Multiple Shapes

MSP (CVPR 2023)

Self-supervised Pre-training with Masked Shape Prediction for 3D Scene Understanding

Unbiased 4D (CVPRW 2023)

Unbiased 4D: Monocular 4D Reconstruction with a Neural Deformation Model

QuAnt (ICLR 2023)

QuAnt: Quantum Annealing with Learnt Couplings

State of the Art in Dense Monocular Non-Rigid 3D Reconstruction.
E. Tretschk^*, N. Kairanda^*, M. B R, R. Dabral, A. Kortylewski, B. Egger, M. Habermann, P. Fua, C. Theobalt and and V. Golyanik.
^* equal contribution.
Eurographics 2023 (Full STARs).
[draft] [project page] [bibtex]

This survey focuses on state-of-the-art methods for dense non-rigid 3D reconstruction of various deformable objects and composite scenes from monocular videos or sets of monocular views. It reviews the fundamentals of 3D reconstruction and deformation modeling from 2D image observations. We then start from general methods—that handle arbitrary scenes and make only a few prior assumptions—and proceed towards techniques making stronger assumptions about the observed objects and types of deformations (e.g. human faces, bodies, hands, and animals). A significant part of this STAR is also devoted to classification and a high-level comparison of the methods, as well as an overview of the datasets for training and evaluation of the discussed techniques. We conclude by discussing open challenges in the field and the social aspects associated with the usage of the reviewed methods.

MHMC (EG 2023)

Scene-Aware 3D Multi-Human Motion Capture from a Single Camera

IMoS (EG 2023)

IMoS: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions

HiFECap (BMVC 2022)

HiFECap: Monocular High-Fidelity and Expressive Capture of Human Performances

QRNG (IEEE Access)

Generation of Truly Random Numbers on a Quantum Annealer

MoCapDeform (3DV 2022)

MoCapDeform: Monocular 3D Human Motion Capture in Deformable Scenes

Quantum Frank-Wolfe (ECCV 2022)

UnrealEgo (ECCV 2022)

UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture

QuMoSeg (ECCV 2022)

Quantum Motion Segmentation

HULC (ECCV 2022)

HULC: 3D HUman Motion Capture with Pose Manifold Sampling and Dense Contact Guidance

NeRF-OSR (ECCV 2022)

NeRF for Outdoor Scene Relighting

Advances in Neural Rendering.
A. Tewari^*, J. Thies^*, B. Mildenhall^*, P. Srinivasan^*, E. Tretschk, Y. Wang, C. Lassner, V. Sitzmann, R. Martin-Brualla, S. Lombardi, C. Theobalt, M. Niessner, J. T. Barron, G. Wetzstein, M. Zollhöfer and V. Golyanik.
^* equal contribution.
State of the Art Report at Eurographics 2022.
[paper] [project page] [bibtex]

This state-of-the-art report on advances in neural rendering focuses on methods that combine classical rendering principles with learned 3D scene representations, often now referred to as neural scene representations. A key advantage of these methods is that they are 3D-consistent by design, enabling applications such as novel viewpoint synthesis of a captured scene. In addition to methods that handle static scenes, we cover neural scene representations for modeling nonrigidly deforming objects and scene editing and composition. While most of these approaches are scene-specific, we also discuss techniques that generalize across object classes and can be used for generative tasks. In addition to reviewing these state-ofthe-art methods, we provide an overview of fundamental concepts and definitions used in the current literature. We conclude with a discussion on open challenges and social implications.

PE (CVPR 2022)

Playable Environments: Video Manipulation in Space and Time