The objective of DYNAMICS is to develop a new methodology for 4D reconstruction of real world scenes with a small number of cameras, as well as to learn statistical models from the captured data sets. A 4D reconstruction refers to a sequence of accurate 3D reconstructions (including geometry, topology and surface properties) of a dynamic (evolving in time) real-world scene. We aim to build a robust lightweight capture system that can be easily installed and used (e.g. in the living room of a house, in outdoor environments, and broadly under various spatial and temporal constraints).
We are developing a novel interactive software system for motion estimation capitalizing on our experience from the predecessor project DENSITY and exploring new directions (new hardware and machine learning methods).
Specifically, the project DYNAMICS can be subdivided into several work packages according to the target scenarios and concerned areas of computer vision:
1) Software for an interactive monocular 4D reconstruction of non-rigid scenes. The main components are modules dealing with non-rigid structure from motion (NRSfM) pipeline and non-rigid registration. Underlying technology will allow to reconstruct non-rigidly deforming scenes with a minimal number of assumptions from a single RGB camera. Target scenarios include endoscopy, capture of facial expressions, small motion and post-factum reconstructions.
2) Software for robust 4D reconstruction from multiple views incorporating optical flow and scene flow with additional assumptions. We plan to assemble a capture studio with five Emergent HT-4000C high-speed cameras (a multi-view setting). Here, we aim at the highest precision and richness of detail in the reconstructions.
3) 3D shape templates with attributes derived from real data using deep learning techniques. The main objective of this work package is to provide statistical models as a prior knowledge in order to increase the robustness and accuracy of reconstructions. Furthermore, the shape templates will allow for more accurate reconstructions of articulated motion (e.g. skeleton poses) from uncalibrated multi-view settings.
DYNAMICS is a BMBF project with an emphasis on development of core technologies applicable in other ongoing and forthcoming projects in the Augmented Vision Lab.
