We present a process for Driving High Resolution Facial Scans with Video Performance Capture with control of viewpoint and illumination. The performance is based on one or more high-quality geometry and reflectance scans of an actor in static poses, driven by one or more video streams of a performance. We compute optical flow correspondences between neighboring video frames, and a sparse set of correspondences between static scans and video frames.
Driving High Resolution Facial Scans with Video Performance Capture
The latter are made possible by leveraging the relight ability of the static 3D scans to match the viewpoint(s) and appearance of the actor in videos taken in arbitrary environments. As optical flow tends to compute proper correspondence for some areas but not others, we also compute a smoothed, per pixel confidence map for every computed flow, based on normalized cross-correlation. These flows and their confidences yield a set of weighted triangulation constraints among the static poses and the frames of a performance.
Given a single artist-prepared face mesh for one static pose, we optimally combine the weighted triangulation constraints, along with a shape regularization term, into a consistent 3D geometry solution over the entire performance that is drift-free by construction. In contrast to previous work, even partial correspondences contribute to drift minimization, for example where a successful match is found in the eye region but not the mouth. Our shape regularization employs a differential shape term based on a spatially varying blend of the differential shapes of the static poses and neighboring dynamic poses, weighted by the associated flow confidences.
These weights also permit dynamic reflectance maps to be produced for the performance by blending the static scan maps. Finally as the geometry and maps are represented on a consistent artist-friendly mesh, we render the resulting high-quality animated face geometry and animated reflectance maps using standard rendering tools.
Skin Micro structure Deformation with Displacement Map Convolution
Technique for synthesizing the effects of skin micro-structure deformation by anisotropically convolving a high-resolution displacement map to match normal distribution changes in measured skin samples. We use a 10-micron resolution scanning technique to measure several in vivo skin samples as they are stretched and compressed in different directions, quantifying how stretching smooths the skin and compression makes it rougher.
Tabulate the resulting surface normal distributions, and show that convolving a neutral skin micro-structure displacement map with blurring and sharpening filters can mimic normal distribution changes and micro-structure deformations. We implement the spatially-varying displacement map filtering on the GPU to interactively render the effects of dynamic micro-geometry on animated faces obtained from high-resolution facial scans.