Real-Time Facial Segmentation\\ and Performance Capture from RGB Input
Shunsuke Saito \qquad Tianye Li \qquad Hao Li
\input{abstract \keywords{real-time facial performance capture, face segmentation, deep convolutional neural network, regression
abstract.tex
We introduce the concept of unconstrained real-time 3D facial performance capture through explicit semantic segmentation in the RGB input. To ensure robustness, cutting edge supervised learning approaches rely on large training datasets of face images captured in the wild. While impressive tracking quality has been demonstrated for faces that are largely visible, any occlusion due to hair, accessories, or hand-to-face gestures would result in significant visual artifacts and loss of tracking accuracy. The modeling of occlusions has been mostly avoided due to its immense space of appearance variability. To address this curse of high dimensionality, we perform tracking in unconstrained images assuming non-face regions can be fully masked out. Along with recent breakthroughs in deep learning, we demonstrate that pixel-level facial segmentation is possible in real-time by repurposing convolutional neural networks designed originally for general semantic segmentation. We develop an efficient architecture based on a two-stream deconvolution network with complementary characteristics, and introduce carefully designed training samples and data augmentation strategies for improved segmentation accuracy and robustness. We adopt a state-of-the-art regression-based facial tracking framework with segmented face images as training, and demonstrate accurate and uninterrupted facial performance capture in the presence of extreme occlusion and even side views. Furthermore, the resulting segmentation can be directly used to composite partial 3D face models on the input images and enable seamless facial manipulation tasks, such as virtual make-up or face replacement.
ack.tex
\section*{Acknowledgements}

We would like to thank Joseph J. Lim, Qixing Huang, Duygu Ceylan, Lingyu Wei, Kyle Olszewski, Harry Shum, and Gary Bradski for the fruitful discussions and the proofreading. We also thank Rui Saito and Frances Chen for being our capture models. This research is supported in part by Adobe, Oculus \& Facebook, Sony, Pelican Imaging, Panasonic, Embodee, Huawei, the Google Faculty Research Award, The Okawa Foundation Research Grant, the Office of Naval Research (ONR) / U.S. Navy, under award number N00014-15-1-2639, the Office of the Director of National Intelligence (ODNI), and Intelligence Advanced Research Projects Activity (IARPA), under contract number 2014-14071600010. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purpose notwithstanding any copyright annotation thereon.
conc.tex
\section{Conclusion}\label{sec:conc}


\begin{wrapfigure}{r}{0.5\textwidth}
\vspace{-0.85cm}
 \centering
\includegraphics[width=0.48\textwidth]{images/results/accuracy}
\caption{Error evaluation on different tracking methods. %The error of expression, rotation and translation are illustrated separately. Gray region in the background represent the fraction of region occluded, as shown on the right side Y axis. Note that a clear segmentation is required to guarantee better performance, and such a performance requires appropriate selection of network architecture as well as augmented training data.
 \label{fig:accuracy}}
\vspace{-0.5cm}
\end{wrapfigure}

We demonstrate that real-time, accurate pixel-level facial segmentation is possible using only unconstrained RGB images with a deep learning approach. Our experiments confirm that a segmentation network with two-stream deconvolution network and shared convolution network is not only critical for extracting both the overall shape and fine-scale details effectively in real-time,
but also presents the current state-of-the-art in face segmentation. We also found that a carefully designed data augmentation strategy effectively produces sufficiently large training datasets for the CNN to avoid overfitting, especially when only limited ground truth segmentations are available in public datasets. In particular, we demonstrate the first successful facial segmentations for skin-colored occlusions such as hands and arms using composited hand datasets on both positive and negative training samples. Consequently, we show that significantly superior tracking accuracy and robustness to occlusion can be achieved by processing images with masked face regions using a state-of-the-art facial performance capture technique~\cite{cao2014displaced}. Training the DDE regressor with images containing only facial regions and augmenting the dataset with synthetic occlusions ensures continuous tracking in the presence of challenging occlusions (e.g., hair and hands). Although we focus on 3D facial performance capture, we believe the key insight of this paper - reducing the dimensionality using semantic segmentation - is generally applicable to other vision problems beyond facial tracking and regression.

%We demonstrate that accurate pixel-level facial segmentations are possible in real-time from only unconstrained RGB images using a deep learning approach. Our experiments confirm that a segmentation network with two-stream deconvolution network and shared convolution network is not only critical for extracting both the overall shape and fine-scale details effectively in real-time,
%but also presents the current state of the art in face segmentation. We also found that a carefully designed data augmentation strategy is effective for producing sufficiently large training datasets for the convolutional neural network to avoid overfitting, especially when only limited ground truth segmentations are available in public datasets. In particular, we demonstrate for the first time, successful facial segmentations for skin-colored occlusions such as hands and arms, using composited hand datasets on both positive and negative training samples. Consequently, we show that significantly superior tracking accuracy and robustness to occlusion can be achieved by processing images with masked face regions using a state of the art facial performance capture technique~\cite{cao2014displaced}. By training the DDE regressor with images that only contain face parts and augmenting the dataset with synthetic occlusions, we can ensure continuous tracking in the presence of challenging occlusions such as hair and hands. Although we focus on 3D facial performance capture, we believe the key insight of this paper - reducing the dimensionality of input using semantic segmentation - can be generally applicable to many other vision problems beyond facial tracking and regression.

%not only regression problems (e.g., facial landmark estimation, pose estimation), but also to 

%While we restrict our explorations within the domain of facial animation, 

%We have shown that deep neural network can very reliably produce pixel-level facial segmentations in fully unconstrained environments, with sufficient data and our improved architecture.
%* In particular deep neural nets can segment occlusions with similar colors to the skin by discovering deep features.
%
%We have shown that part-based regression is effective in fitting a 3D model given partial input data based on a clean masked RGB input. This approach only works well if the segmentation is sufficiently good.
%
%What is crucial?
%
%Impact:
%* our system shows that robust tracking is possible in the presence of strong occlusions, which was not posisble before especially due to hairstyles etc. we can allow capture during natural conversations (with hand gestures etc.)

\paragraph{Limitations and Future Work.}

Though surpassing the state-of-the-art, our solution is far from perfect. Since only limited training data is used, the resulting segmentation masks can still yield flickering boundaries. We wish to explore the use of a temporal information, as well as the modeling of domain-specific priors to better handle lighting variations. In addition to facial regions, we would also like to extend our ideas to segment other body parts to facilitate more complex compositing operations that include hands, bodies, and hair.  

%We would also like to segment not only facial regions but extend our ideas also fore-and background
%
%
%We would like to segment not only facial regions but also front and background, so that we can visualize hands in front of a digital mask with different shapes than the tracked template, which is an important step for AR applications
%
%We want to capture fine details over multiple frames and reconstruct them behind occlusions based on the our segmentation framework. Can we capture entire human faces based on unconstrained collected data from partial views?
%
%This work focusses on the robustness and segmentation part, we also wish to achieve higher fidelity of identity and expressions. Coupling this with large labeled datasets.

eccv2016submission.tex
% last updated in April 2002 by Antje Endemann
% Based on CVPR 07 and LNCS, with modifications by DAF, AZ and elle, 2008 and AA, 2010, and CC, 2011; TT, 2014; AAS, 2016

\documentclass[runningheads]{llncs}
\usepackage{graphicx}
\usepackage{amsmath,amssymb} % define this before the line numbering.
%\usepackage{ruler}
\usepackage{color}
\input{pre}
\usepackage[width=122mm,left=12mm,paperwidth=146mm,height=193mm,top=12mm,paperheight=217mm]{geometry}
\usepackage{caption}
\usepackage{hyperref}
\hypersetup{
     colorlinks   = true,
     citecolor    = black,
     linkcolor = black,
    urlcolor=black
}

\linespread{0.98}

\begin{document}
%\DeclareMathOperator*{\argmin}{\arg\!\min}
%\renewcommand\thelinenumber{\color[rgb]{0.2,0.5,0.8}\normalfont\sffamily\scriptsize\arabic{linenumber}\color[rgb]{0,0,0}}
%\renewcommand\makeLineNumber {\hss\thelinenumber\ \hspace{6mm} \rlap{\hskip\textwidth\ \hspace{6.5mm}\thelinenumber}}
%\linenumbers
\pagestyle{headings}
%\mainmatter
\def\ECCV16SubNumber{1622}  % Insert your submission number here

\title{Real-Time Facial Segmentation\\ and Performance Capture from RGB Input} % Replace with your title

\titlerunning{Real-Time Facial Segmentation and Performance Capture from RGB Input}

\authorrunning{Shunsuke Saito, Tianye Li, Hao Li}

\author{Shunsuke Saito \qquad Tianye Li \qquad Hao Li}
\institute{Pinscreen \qquad University of Southern California}


\maketitle

\begin{abstract}
\input{abstract}
\keywords{real-time facial performance capture, face segmentation, deep convolutional neural network, regression}
\end{abstract}

\input{intro}
\input{related}
%\input{problem_statement}
\input{system}
\input{result}
\input{conc}
\input{ack}

%\clearpage

\bibliographystyle{splncs}
\bibliography{egbib}
\end{document}

intro.tex
\section{Introduction}\label{sec:intro}

Recent advances in real-time 3D facial performance capture~\cite{weise2011realtime,Bouaziz:2013:OMR,li2013realtime,Cao:2013:SRR,cao2014displaced,cao2015real,hsieh2015unconstrained} have not only transformed the entertainment industry with highly scalable animation and affordable production tools~\cite{Faceshift:2014:FS}, but also popularized mobile and social media apps with facial manipulation and analytics software. %Soon we are likely to see expansion of this capture technology to exciting VR and AR applications, as hinted by several recent work~\cite{li2015facial,thies2015real}. 
The key factors behind this democratization are: (1) the ability to capture compelling facial animations in real-time and (2) the accessibility of commodity sensors (video and RGB-D). Many state-of-the-art techniques have been developed to operate robustly in natural environments, but pure RGB solutions are still susceptible to occlusions (e.g., caused by hair, hand-to-face gestures, or accessories), which result in unpleasant visual artifacts or the inability to correctly initialize facial tracking. 

While it is known that the shape and appearance of fully visible faces can be represented compactly through linear models~\cite{Blanz:1999:MMS,cao2014facewarehouse}, any occlusion or uncontrolled illumination could cause high non-linearities to a 3D face fitting problem. As this space of variation becomes intractable, supervised learning methods have been introduced to predict facial shapes through large training datasets of face images captured under unconstrained and noisy conditions. We observe that if such \emph{occlusion noise} can be fully eliminated, the dimensionality of facial modeling could be drastically reduced to that of a well-posed and constrained problem. In other words, if reliable dense facial segmentation  is possible, 3D facial tracking from RGB input becomes a significantly easier problem. Only recently has the deep learning community demonstrated highly effective semantic segmentations, such as the fully convolutional network (FCN) of~\cite{long_shelhamer_fcn} or the deconvolutional network (DeconvNet) of~\cite{noh2015learning}, by repurposing highly efficient classification networks~\cite{Chatfield14,Krizhevsky_imagenetclassification} for dense predictions of general objects (e.g., humans, cars, etc.). %While, leveraging these techniques for fac

We present a real-time facial performance capture approach by explicitly segmenting facial regions and processing masked RGB data. 
We rely on the effectiveness of deep learning to achieve clean facial segmentations in order to enable robust facial tracking under severe occlusions.
%A straightforward adoption of existing convolutional neural networks with segmented face images as training input would yield either too coarse~\cite{long_shelhamer_fcn} or too noisy results~\cite{noh2015learning}. While Noh and coworkers~\cite{noh2015learning} have suggested the combination of both DeconvNet~\cite{noh2015learning} (for fine details) and FCN~\cite{long_shelhamer_fcn} (for overall shape) output probability maps via conditional random fields, computing the two networks separately is prohibitively expensive for our real-time purposes. 
We propose an end-to-end segmentation network that also uses a two-stream deconvolution network with complementary characteristics, but shares the lower convolution network to enable real-time performance. A final convolutional layer recombines both outputs into a single probability map which is converted into a refined segmentation mask via graph cut algorithm~\cite{Rother:2004:GIF}.
Our 3D facial tracker is based on a state-of-the-art displaced dynamic expression (DDE) method~\cite{cao2014displaced} trained with segmented input data.
%We ensure real-time performance while 
%
%, we combine the robustness of the fully connected network (FCN) of~\cite{long_shelhamer_fcn} and the 
%
%A deep convolutional neural network for 
%combine the use of two state of the art  networks for 
%Since a straightforward adoption of existing convolutional neural networks with segmented face images as training input would yield 
%, the output is either too coarse~\cite{long_shelhamer_fcn} or too noisy~\cite{noh2015learning}. 
%We introduce an intrinsically fused deconvolutional network (IF-DeconvNet) for real-time facial segmentation which combines the robust FCN~\cite{long_shelhamer_fcn} and DeconvNet~\cite{noh2015learning}, which recovers fine details, into a single end-to-end segmentation network. 
While the network parameters of pre-learned representations are transferred to our facial segmentation task, we fine tune the network using training samples from the LFW~\cite{LFWTech} and FaceWarehouse~\cite{cao2014facewarehouse} datasets. Furthermore, separating facial regions from occluding objects with similar colors and fine structures (e.g. hands) is still extremely challenging for any existing segmentation network, since no such supervision is provided by existing data sets. We propose a data augmentation strategy based on perturbations, croppings, occlusion generation, hand compositings, as well as the use of negative samples containing no faces. Once our dense prediction model is trained, we replace the training database for DDE regression with masked faces obtained from our convolutional network. 


%%\ramen{So as to discover face regions in a downsized image while retaining real-time performance, we present a intrinsically fused deconvolution network (IF-DeconvNet) by combining a robust segmentation network, fully convolutional neural network (FCN)~\cite{long_shelhamer_fcn} and a detail-preserved segmentation network, DeconvNet~\cite{noh2015learning} as a single network. In contrast to the naive ensemble approach EDeconvNet proposed in~\cite{noh2015learning}, our network is more efficient and achieve better performance. Furthermore, we demonstrate how our dedicated training processes and data augmentation strategies including cropping and occlusion generations, as well as negative hand image samples, boost the segmentation accuracy using training samples from the LFW dataset~\cite{LFWTech}, FaceWarehouse~\cite{cao2014facewarehouse}. The probability map generated by DCNN is used as unary terms in a post-hoc refinement solution based on graph cut segmentation. Once our dense prediction model is trained, we replace the training database for DDE regression with masked face samples obtained from our convolutional network. } 

%We demonstrate how a fully convolutional neural network~\cite{long_shelhamer_fcn} can be trained to effectively discover face regions in a highly downsized image while retaining real-time performance. Instead of adopting an end-to-end segmentation framework such as DeconvNet~\cite{noh2015learning}, we propose a deterministic but efficient post-hoc refinement solution based on graph cut segmentation on superpixels. While the network parameters of pre-learned representations are transferred to our facial segmentation task, we fine tune the network using training samples from the LFW dataset~\cite{LFWTech}, FaceWarehouse~\cite{cao2014facewarehouse}, and important data augmentation strategies including cropping and occlusion generations, as well as negative hand image samples. Once our dense prediction model is trained, we replace the training database for DDE regression with masked face samples obtained from our convolutional network. 

Our approach retains every capability of Cao et al.~\cite{cao2014displaced}'s algorithm such as real-time performance and the absence of a calibration process, but considerably enhances its robustness w.r.t. occlusions and even side views. We demonstrate uninterrupted tracking in the presence of highly challenging occlusions such as hands which have similar skin tones as the face and fine scale boundary details. Furthermore, our facial segmentation solution provides masked images which enables interesting compositing effects such as tracked facial models under hair and other occluding objects. These capabilities were only demonstrated recently using a robust geometric model fitting approach on depth sensor data~\cite{hsieh2015unconstrained}. Since we only assume RGB data as input, our method not only addresses a fundamental challenge of real-time facial segmentation, but also provides unmatched flexibility for deployment, and requires minimal implementation effort.

\vspace{0.5cm}

We make the following contributions:
\begin{itemize}
	\item We present the first real-time facial segmentation framework from pure RGB input using a convolutional neural network. We demonstrate the importance of carefully designed datasets and data augmentation strategies for handling challenging occlusions such as hands.
	\item We improve the efficiency and accuracy of existing segmentation networks using an architecture based on two-stream deconvolution networks and shared convolution network.
	\item We demonstrate superior tracking accuracy and robustness through explicitly facial segmentation and regression with masked training data, and outperform the current state-of-the-art.
\end{itemize}

%Real-time performance  
%
%
%To ensure real-time performance, we 
%
%
%
% is the 
%
% into 
%
%Applications/Vision:
%* virtual makeup, face beautification, replacement, while segmenting occlusions
%* show an example of facial hair (beard demo)
%* virtual glasses
%* driving avatars in unconstrained environments, robustness to hand gestures and hair occlusions
%* analytics from videos -> personalized advertising in Minority Report
%* some companies are already experimenting with facial recognition to tailor advertisting based on gender age information.
%* 3D face recognition in the wild based as a convenient way for payment.
%* mobile applications such as Snapchat Lenses
%* interest in AR and VR to digitize avatars with our appearances.

%Objective:
%* from RGB input, provide uninterrupted facial tracking and explicit pixel-level facial segmentation
%
%
%State of the art
%* detailed real-time facial performance capture in the wild is possible
%* RGB (no occlusion handling or segmentation) vs depth sensor has been recently enabled, but depth sensors are significantly less deployed and online videos do not have depth information, also depth sensors are sensitive to daylight, so they do not perform well outdoors under sunlight, existing systems have relatively low effective resolution.
%
%Problem/Challenge:
%* Wide appearances variations of non-face objects (different hairstyle, hand shapes, objects, illumination environment) makes it difficult to model non-face regions)
%* what if object has similar color as face?
%
%* The robustness of existing facial perfamnce capture techniques rely on facial features of the full face, difficult to provide continuous input if certain regions are occluded.
%* computation has to be real-time
%
%Approach:
%* we develop a deep learning framework to segment facial regions
%* we propose a robust part-based regression technique to handle incomplete input data.
%
%Advantages/Features:
%
%* Our method can be deployed on existing mobile devices
%* GPU intensive computation using deep neural network can be offloaded wirelessly through a remote server
%* as with other state of the art techniques
%* No calibration required, instant user switching
%* Identity and expressions are learned online
%
%Results and Applications
%* we demonstrate challenging occlusion examples including, hands which have skin colors and fine-scale boundary details, hair and a wide range of props such as glasses.
%* we show uninterrupted tracking even in the presence of extremely large occlusions of the face.
%* we show that our part-based regression method is on-par with existing full-face regression techniques
%* we demonstrate make-up and face replacement applications and AR applications with digital avatars.
pre.tex
% Nice table
\usepackage[table]{xcolor}
\usepackage{tabularx}
\usepackage{booktabs}
\usepackage{pdfcomment}
\usepackage{color}
\usepackage{bm}
\definecolor{lightbluishgrey}{rgb}{0.9,0.91,0.95}
\newcommand{\ra}[1]{\renewcommand{\arraystretch}{#1}}
\newcommand{\NO}{{\color{red}\textsf{X}}}
\newcommand{\YES}{$\bullet$}
% Comments
\newcommand{\obsolete}[1]{{\textcolor[rgb]{0.5,0.5,0.5}{#1}}}
\newcommand{\todo}[1]{{\bf\textcolor{red}{TODO: #1}}}
\newcommand{\ramen}[1]{{\textcolor[rgb]{1.0,0.0,0.0}{#1}}}
\newcommand{\Hao}[1]{{\bf\textcolor[rgb]{0.9,0.5,0.0}{Hao: #1}}}
\hypersetup{citecolor=black}
\newcommand{\cheatvspace}[1]{}
\newcommand{\newhl}[1]{#1}
\newcommand{\newnewhl}[1]{#1}
\newcommand{\proj}{{\cal P}}


\usepackage{wrapfig}
%%% To use cached, low resolution figures (for a smaller output pdf)
%%% Run:
%%%   mogrify -path lowresfig/ -compress jpeg -density 150 fig/*.pdf
%%% And uncomment this:
%\newcommand{\fig}{lowresfig}
%\usepackage{background}
%\backgroundsetup{%
%  scale=1,       %% change accordingly
%  angle=0,       %% change accordingly
%  opacity=.2,    %% change accordingly
%  color =red,  %% change accordingly
%  contents={{\fontsize{50}{60}\selectfont LOW RES FIGURES}}
%}
%% And comment this:
\newcommand{\fig}{fig}


\newcommand{\refequ}[1] {Equation~(\ref{equ:#1})}
\newcommand{\reffig}[1] {Figure~\ref{fig:#1}}
\newcommand{\refoutsidefig}[1] {Figure~#1}
\newcommand{\reftab}[1] {Table~\ref{tab:#1}}
\newcommand{\reflem}[1] {Lemma~\ref{lem:#1}}
\newcommand{\refsec}[1] {Section~\ref{sec:#1}}
\newcommand{\refapp}[1] {Appendix~\ref{app:#1}}


% Notation
% Add decoration to indicate rest-state
\newcommand{\rest}[1]{\widebar{#1}}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% MATH (should come at end of diary.cls)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% General
\let\mat = \mathbf
\usepackage{xfrac}
\newcommand{\onehalf}{\sfrac{1}{2}}
% Need this for cases
\usepackage{amsmath}
\usepackage{amssymb}    % need for things like varnothing
\usepackage{cancel}
%http://tex.stackexchange.com/a/53829/13600
\usepackage[T1]{fontenc}
% Specialized symbols ranging from general to specific
\newcommand*{\dittostraight}{\textquotesingle\textquotesingle}
\let\Lap = \Delta
\let\Grad = \nabla
\newcommand{\R}{\mathbb{R}}
\newcommand{\vc}[1]{\mathbf{#1}}
\newcommand{\C}{\mat{C}}
\newcommand{\Crest}{\rest{\C}}
\newcommand{\transpose}{{\mathsf T}}
% directional derivative
\newcommand{\dd}[2]{\frac{\partial#1}{\partial#2}}
% Defined on/Defined over: x|_{∂Ω}
\newcommand{\defon}[2]{\left.{#1}\right|_{#2}}
% **(RE)DEFINE** all \a to be \vc{a}. In the comments I reveal what I'm
% overwriting. Because of this you're likely to get mysterious errors when
% typesetting foreign words:
%
% ! LaTeX Error: \mathbf allowed only in math mode.
%
% Seems like this is used to create an accent mark \a'
\renewcommand{\a}{\vc{a}}
% Used to put a little bar under a letter
\renewcommand{\b}{\vc{b}}
% Used to spell fa\c{c}ade
\renewcommand{\c}{\vc{c}}
% Used to put a little dot under a letter
\renewcommand{\d}{\vc{d}}
\newcommand{\e}{\vc{e}}
\newcommand{\f}{\vc{f}}
\newcommand{\g}{\vc{g}}
\newcommand{\h}{\vc{h}}
% Used to make Dotless i like in Turkish
\renewcommand{\i}{\vc{i}}
% Used to make Dotless i like in Turkish
\renewcommand{\j}{\vc{j}}
% Used to spell Polish and Native American words
\renewcommand{\k}{\vc{k}}
% Used to draw an l with a stroke (Polish?)
\renewcommand{\l}{\vc{l}}
\newcommand{\m}{\vc{m}}
\newcommand{\n}{\vc{n}}
% Used to place a little
\renewcommand{\o}{\vc{o}}
\newcommand{\p}{\vc{p}}
\newcommand{\q}{\vc{q}}
% Used to put a little "ring" ontop of a letter like \r{a} = \aa --> å
\renewcommand{\r}{\vc{r}}
\newcommand{\s}{\vc{s}}
% Used to put a "tie" (inverted U) over two letters
\renewcommand{\t}{\vc{t}}
% Little u over a letter
\renewcommand{\u}{\vc{u}}
% Used for putting a little upside down hat on Slavic words \v{Z}eljko
%\renewcommand{\v}{\vc{v}}
\newcommand{\vv}{\vc{v}}
\newcommand{\w}{\vc{w}}
\newcommand{\x}{\vc{x}}
\newcommand{\y}{\vc{y}}
\newcommand{\z}{\vc{z}}
\newcommand{\A}{\mat{A}}
\newcommand{\B}{\mat{B}}
%\renewcommand{\C}{\mat{C}}
\newcommand{\D}{\mat{D}}
\newcommand{\E}{\mat{E}}
\newcommand{\F}{\mat{F}}
\newcommand{\G}{\mat{G}}
\renewcommand{\H}{\mat{H}}
\newcommand{\Hrest}{\rest{\H}}
\newcommand{\I}{\mat{I}}
\newcommand{\J}{\mat{J}}
\newcommand{\K}{\mat{K}}
\newcommand{\Kz}{\overline{\mat{K}}}
\renewcommand{\L}{\mat{L}}
\newcommand{\M}{\mat{M}}
\newcommand{\N}{\mat{N}}
%\newcommand{\O}{\mat{O}}
\renewcommand{\P}{\mat{P}}
\newcommand{\RR}{\mat{R}}
\renewcommand{\S}{\mat{S}}
\newcommand{\T}{\mat{T}}
\newcommand{\U}{\mat{U}}
\newcommand{\V}{\mat{V}}
\newcommand{\Vrest}{\rest{\V}}
\newcommand{\1}{\mat{1}}
\newcommand{\W}{\mat{W}}
\newcommand{\X}{\mat{X}}
\newcommand{\Y}{\mat{Y}}
\newcommand{\Z}{\mat{Z}}
\newcommand{\argmin}{\mathop{\text{argmin }}}
\newcommand{\ii}{\vc{i}}
\newcommand{\jj}{\vc{j}}
\newcommand{\kk}{\vc{k}}
\newcommand{\ft}{\tilde{f}}
\newcommand{\cotaa}{\cot{\alpha_a}}
\newcommand{\cotab}{\cot{\alpha_b}}
\newcommand{\cotac}{\cot{\alpha_c}}
\newcommand{\cotba}{\cot{\beta_a}}
\newcommand{\cotbb}{\cot{\beta_b}}
\newcommand{\cotbd}{\cot{\beta_d}}
\newcommand{\cotaone}{\cot{\alpha_1}}
\newcommand{\cotatwo}{\cot{\alpha_2}}
\newcommand{\cotbone}{\cot{\beta_1}}
\newcommand{\cotbtwo}{\cot{\beta_2}}
% unit normal
\newcommand{\un}{\hat{\vc{n}}}



% scientific notation
% http://tex.stackexchange.com/a/70532/13600
\usepackage{siunitx}
\sisetup{output-exponent-marker=\text{e}, bracket-negative-numbers,
open-bracket={\text{-}}, close-bracket={}}

\usepackage[mathletters]{ucs}
\usepackage[utf8x]{inputenc}
% μ
\usepackage{upgreek}
\usepackage{dblfloatfix}
\newcommand{\microsec}{\upmu{}\text{s}}



% FIX LINE NUMBERS AFTER EQUATION
% http://changilkim.wordpress.com/2013/05/13/siggraph-latex-template-line-number-correction-hack/
%\newcommand*\patchAmsMathEnvironmentForLineno[1]{%
%\expandafter\let\csname old#1\expandafter\endcsname\csname #1\endcsname
%\expandafter\let\csname oldend#1\expandafter\endcsname\csname end#1\endcsname
%\renewenvironment{#1}%
%{\linenomath\csname old#1\endcsname}%
%{\csname oldend#1\endcsname\endlinenomath}}%
%\newcommand*\patchBothAmsMathEnvironmentsForLineno[1]{%
%\patchAmsMathEnvironmentForLineno{#1}%
%\patchAmsMathEnvironmentForLineno{#1*}}%
%\AtBeginDocument{%
%\patchBothAmsMathEnvironmentsForLineno{equation}%
%\patchBothAmsMathEnvironmentsForLineno{align}%
%\patchBothAmsMathEnvironmentsForLineno{flalign}%
%\patchBothAmsMathEnvironmentsForLineno{alignat}%
%\patchBothAmsMathEnvironmentsForLineno{gather}%
%\patchBothAmsMathEnvironmentsForLineno{multline}%
%\patchBothAmsMathEnvironmentsForLineno{eqnarray}%
%}

%  bold paragraph header without space after
\newcommand{\nospaceparagraph}[1]{{\sffamily\textbf{#1}}}
\renewcommand{\Re}{\operatorname{Re}}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% IMAGES, GRAPHICS, FIGURES AND CAPTIONS
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\usepackage{graphicx}
\newcommand{\figures}{figures}
\usepackage{wrapfig}
\newcommand{\wf}[6]{%
  \begin{wrapfigure}[#1]{#2}{#3}%
  \centering%
  \includegraphics[#4]{#5}
  #6%
\end{wrapfigure}}
\newcommand{\ig}[2][]{\includegraphics[#1]{#2}}
%\newcommand{\fig}{2}{%
%\begin{figure}%
%\centering%
%#1%
%\caption{#2}%
%\end{figure}}


% WIDE BAR
% http://tex.stackexchange.com/a/60253/13600
\makeatletter
\let\save@mathaccent\mathaccent
\newcommand*\if@single[3]{%
  \setbox0\hbox{${\mathaccent"0362{#1}}^H$}%
  \setbox2\hbox{${\mathaccent"0362{\kern0pt#1}}^H$}%
  \ifdim\ht0=\ht2 #3\else #2\fi
  }
%The bar will be moved to the right by a half of \macc@kerna, which is computed by amsmath:
\newcommand*\rel@kern[1]{\kern#1\dimexpr\macc@kerna}
%If there's a superscript following the bar, then no negative kern may follow the bar;
%an additional {} makes sure that the superscript is high enough in this case:
\newcommand*\widebar[1]{\@ifnextchar^{{\wide@bar{#1}{0}}}{\wide@bar{#1}{1}}}
%Use a separate algorithm for single symbols:
\newcommand*\wide@bar[2]{\if@single{#1}{\wide@bar@{#1}{#2}{1}}{\wide@bar@{#1}{#2}{2}}}
\newcommand*\wide@bar@[3]{%
  \begingroup
  \def\mathaccent##1##2{%
%Enable nesting of accents:
    \let\mathaccent\save@mathaccent
%If there's more than a single symbol, use the first character instead (see below):
    \if#32 \let\macc@nucleus\first@char \fi
%Determine the italic correction:
    \setbox\z@\hbox{$\macc@style{\macc@nucleus}_{}$}%
    \setbox\tw@\hbox{$\macc@style{\macc@nucleus}{}_{}$}%
    \dimen@\wd\tw@
    \advance\dimen@-\wd\z@
%Now \dimen@ is the italic correction of the symbol.
    \divide\dimen@ 3
    \@tempdima\wd\tw@
    \advance\@tempdima-\scriptspace
%Now \@tempdima is the width of the symbol.
    \divide\@tempdima 10
    \advance\dimen@-\@tempdima
%Now \dimen@ = (italic correction / 3) - (Breite / 10)
    \ifdim\dimen@>\z@ \dimen@0pt\fi
%The bar will be shortened in the case \dimen@<0 !
    \rel@kern{0.6}\kern-\dimen@
    \if#31
      \overline{\rel@kern{-0.6}\kern\dimen@\macc@nucleus\rel@kern{0.4}\kern\dimen@}%
      \advance\dimen@0.4\dimexpr\macc@kerna
%Place the combined final kern (-\dimen@) if it is >0 or if a superscript follows:
      \let\final@kern#2%
      \ifdim\dimen@<\z@ \let\final@kern1\fi
      \if\final@kern1 \kern-\dimen@\fi
    \else
      \overline{\rel@kern{-0.6}\kern\dimen@#1}%
    \fi
  }%
  \macc@depth\@ne
  \let\math@bgroup\@empty \let\math@egroup\macc@set@skewchar
  \mathsurround\z@ \frozen@everymath{\mathgroup\macc@group\relax}%
  \macc@set@skewchar\relax
  \let\mathaccentV\macc@nested@a
%The following initialises \macc@kerna and calls \mathaccent:
  \if#31
    \macc@nested@a\relax111{#1}%
  \else
%If the argument consists of more than one symbol, and if the first token is
%a letter, use that letter for the computations:
    \def\gobble@till@marker##1\endmarker{}%
    \futurelet\first@char\gobble@till@marker#1\endmarker
    \ifcat\noexpand\first@char A\else
      \def\first@char{}%
    \fi
    \macc@nested@a\relax111{\first@char}%
  \fi
  \endgroup
}
\makeatother

problem_statement.tex
\section{Problem Statement}\label{sec:problem}

\begin{figure*}[t!]
 \centering
\includegraphics[width=\textwidth]{images/overview/overview}
\caption{Here is an overview of our system
 \label{fig:overview}}
\end{figure*}


\begin{figure}[h!]
 \centering
\includegraphics[width=\columnwidth]{images/overview/system.pdf}
\caption{Capture Setting.
 \label{fig:system}}
\end{figure}


% go through idea here, need to be short, concise, beautiful

* We would like to state the idea to solve unconstrained facial capturing in a more general way, as the problem share common setting with other applications like detection, tracking etc.

Generally, facial applications including detection, tracking, and landmark alignment require algorithms to not only predict facial features e.g. shape positions or expression, but also have a sophisticated system to robustly handle the variation of appearance under unconstrained situation.

In fact, since shape and motion of face can be reliably determined using very few parameters, most of the variation of input comes from objects non-directly related to the face itself, as known as background or occlusion depending on their relative position towards the face.
Since the appearance of such regions is independent with and cannot provide extra guidance to the face, improving facial capture performance by first segmenting and removing unrelated region becomes an intuitive idea.

While the idea itself is simple, the idea beneath this optimization should draw more concern as it reveals a more general methodology and can be applied to improve large amount of existing algorithm handling performance capturing allowing wild environment with potential occlusion.

To recover interesting information of the face, considered as an specific object in the scene of the input image, the variance of all other objects, including light source, camera or other objects obviously cannot help, and should not influence the process.
Although some environment factors e.g. lighting or motion of camera will affect the appearance of one's face, and they are coupled with the parameter of face in the input image within the region of interest, they can be described using simple models and are already well examined (white balance, debluring, denoising etc).

The main challenge lays on the existence of other objects and their highly variant appearance, which by principle cannot be modeled (because arbitrary things can exist at the background).
And as most of occluding objects are not transparent, the regions occupied by them cannot provide any further information to the face.
Note that occluded regions are different from image noise because of inaccurate measurement, e.g. motion blur or exposure noise.
Noisy observation are derived from the ground truth, thus information can still be recovered or reliably estimated given the model of the occurrence of noise.
Occlusions, however, is independent from the desired information.

Therefore, reliably masking out such regions will greatly reduce the variance of input data, without losing any potential information. Existing methods can hence benefit from a much simpler range of input. Since the appearance of background and occlusion are unpredictable, this is also the only possible way to minimize the influence brought by change of such regions.

In fact, with reliable face segmentation, accuracies of existing methods~\cite{cao2014displaced} can be improved as shown in Sec~\ref{sec:result}.
This indicates that although such methods claim to handle variation including occlusion, the intrinsic process of it didn't succeed in eliminating the effect from unrelated objects better than a separate segmentation step does.

Although the segmentation between the face region and unrelated ones contains more indeterminism, a rough region estimation can be obtained quickly with modern GPU and the concept of convolutional neural network in image processing. With further efficient and deterministic (non-learning) refinement we show that an accurate face segmentation in real-time is possible, making this step of image processing ``cost nothing'' and has its potential to become a standard data preprocessing step before any facial capture technique.

% hardness of this problem
%* The segmentation and face capture is a chick and egg problem.
%While the capture problem is more concrete and the segmentation seems to be harder to be solved, its wild application draws people's attention and there are works in the computer vision attacking general image segmentation task quite successfully.
%The segmentation challenge such as in ILSVRC focus on general segmentation between different categories of objects in wild, however to segment only the facial part our of the whole human haven't been studied thoroughly yet.






% here are words from Shunsuke, need reorganize
%Regression has been widely adopted by facial applications such as detection, tracking, and landmark alignment. Given training data, regression-based approaches learn a direct mapping function $\Phi$ from image appearance $\F$ to the target output $\P$ (e.g., landmark locations, 3D shape, or blend shape coefficients). In the unconstrained situation, however, the input image appearance may contain non-face objects front or behind faces. Since arbitrary objects can be present in the image, occluding objects and background can be regarded as noise. Recently researchers have attempted to improve the robustness of regression by cascading weak regressors based on large training data, it is still difficult to model occlusion due to its immense space of appearance variability. Thus, instead, we propose to reduce appearance variability in the images by masking out non-face regions as if the input image only contains a face in both training and inference processes. In this paper, by applying it to facial performance capture, we demonstrate this observation drastically improves the robustness In Sec~\ref{sec:system}, we explain the overview of our system architecture enabling high-fidelity face/non-face segmentation and robust facial performance capture in realtime. We will describe how we obtain segmentation mask in order to achieve robust cascade regression (Sec~\ref{sec:probmap}) and how we incorporate resulting segmentation into regression-based facial tracking (Sec~\ref{sec:tracking}). Sec~\ref{sec:result} demonstrait that our method outperform other state-of-the art methods and achives robust segmentation and face tracking in the challenging cases such as occlusion and large rotation.
%\Shun{TODO: rephrase noise in this section}
related.tex
\section{Related Work}\label{sec:related}

The fields of facial tracking and animation have undergone a long thread of major research milestones
in both, the vision and graphics community, as well as influencing the industry widely over the past two decades.

In high-end film and game production, performance-driven techniques are commonly 
used to scale the production of realistic facial animation. An overview is discussed in 
Pighin and Lewis~\cite{PighinLewisCourse06}. 
To meet the high quality bars, techniques for production
typically build on sophisticated sensor equipments and controlled capture settings~\cite{Guenter:1998:MF,zhang-siggraph2004-stfaces,FurukawaP09,Li:2009:RSV,Bee11,fyffe2011comprehensive,Bhat:2013:HFF,Fyffe:2014:DHF}. While exceptional tracking accuracy can be achieved, these methods are generally computationally expensive and the full visibility of the face needs to be ensured.

On the other extreme, 2D facial tracking methods that work in fully unconstrained settings have been explored extensively for applications such as face recognition and emotion analytics. Even though only sparse 2D facial landmarks are detected, many techniques are designed to be robust to uncontrolled poses, challenging lighting conditions, and rely on a single-view 2D input. Early algorithms are based on parametric models~\cite{li93threed,bregler1994surface,Black:1995:TRR,Essa:1996:MTI,Decarlo:2000:OFC}, but later outperformed by more robust and real-time data-driven methods such as active appearance models (AAM)~\cite{cootes2001active} and constrained local models (CLM)~\cite{Cristinacce:2008:AFL}. While the landmark mean-shift approach of~\cite{Saragih:2011:DMF} and the supervised descent method of~\cite{xiong2013supervised} avoid the need of user-specific training, more efficient solutions exist based on explicit shape regressions~\cite{export:206636,kazemi2014one,ren2014face}. However, these methods are all sensitive to occlusions and only a limited number of 2D features can be detected. 

Weise and colleagues~\cite{weise2009face} demonstrated the first system to produce compelling facial performance capture in real-time using a custom 3D depth sensor based on structured light. The intensive training procedure was later reduced significantly using an example-based algorithm developed by Li and collaborators~\cite{li2010example}. With consumer depth sensors becoming mainstream (e.g., Kinect, Realsense, etc.), a whole line of real-time facial animation research have been developed with focus on deployability. The work of~\cite{weise2011realtime} incorporated pre-recorded motion priors to ensure stable tracking for noisy depth maps, which resulted in the popular animation software, Faceshift~\cite{Faceshift:2014:FS}. By optimizing the identity and expression models online, Li and coworkers~\cite{li2013realtime}, as well as Bouaziz and collaborators~\cite{Bouaziz:2013:OMR} eliminated the need of user-specific calibration. For uninterrupted tracking under severe occlusions, Hsieh and colleagues~\cite{hsieh2015unconstrained} recently proposed an explicit facial segmentation technique based on depth and RGB cues. While the idea of explicitly segmenting faces is similar to our work, their method relies on depth sensor input.

While the generation of 3D facial animations from pure RGB input have been demonstrated using sparse 2D landmarks detection~\cite{Pighin:etal:1999,Chuang02Performance,Chai:2003:VCF}, a superior performance capture fidelity and robustness has only been shown recently by Cao and coworkers~\cite{Cao:2013:SRR} using a 3D shape regression approach. Cao and colleagues~\cite{cao2014displaced} later extended the efficient two-level boosted regression technique introduced in~\cite{export:206636} to the 3D case in order to avoid user-specific calibration. Higher fidelity facial tracking from monocular video has also been demonstrated with additional high-resolution training data~\cite{cao2015real}, very large datasets of a person~\cite{totalmoving}, or more expensive non-real-time computation~\cite{garrido2013reconstructing,shi2014automatic}. While robust to unconstrained lighting environments and large head poses, these methods are sensitive to large occlusions and cannot segment facial regions.% as with depth sensors~\cite{hsieh2015unconstrained}. 
%Consequently, occlusions caused by hairstyles and hand gestures must be avoided, and many facial manipulation tasks such as face replacement require fully visible faces~\cite{Dale:2011:VFR,GVRTPT14,thies2015real}.

Due to the immense variation of facial appearances in unconstrained images, it is extremely challenging to obtain clean facial segmentations at the pixel level.  
The hierarchical CNN-based parsing network of Luo and collaborators~\cite{luo2012hierarchical} generates masks of individual facial components such as eyes, nose, and mouth even in the presence of occlusions, but does not segment the facial region as a whole. Smith and coworkers~\cite{smith2013exemplar} use an example-based approach for facial region and component segmentation, but the method requires sufficient visibility of the face. These two methods are computationally intensive and susceptible to wrong segmentations when occlusions have similar colors as the face. By alternating between face mask prediction and landmark localization with deformable part models, Ghiasi and Fowlkes~\cite{ghiasi2015using} have recently demonstrated state-of-the-art facial segmentation results on the Caltech Occluded Faces in the Wild (COFW) dataset~\cite{burgos2013robust} at the cost of expensive computations. 
Without explicitly segmenting the face, occlusion handling methods have been proposed for the detection of 2D landmarks within an AAM frameworks~\cite{Gross:2006:AAM}, but superior results were later shown using techniques based on discriminatively trained deformable parts model ~\cite{Ramanan:2012:FDP,ghiasi2014occlusion}. Highly efficient landmark detection has been recently demonstrated using cascade of regressors trained with occlusion data~\cite{burgos2013robust,YuLBM14}. 

%In contrast to these methods, that aim at detecting sparse 2D landmarks robustly in the presence of occlusions, we explicitly compute a pixel-level face segmentation in real-time using a deep learning approach, and incorporate the resulting binary mask as prior into our regression-based dense 3D face tracker. While state of the art semantic segmentation networks have demonstrated impressive performances on general objects (e.g., humans, cars, etc.)~\cite{long_shelhamer_fcn,chen2014semantic,zheng2015conditional,liu2015semantic} in challenging unconstrained images, the robust and real-time segmentation of detailed facial regions remains unexplored.






%Weise and colleagues~\cite{weise2009face} demonstrated the first system to produce compelling facial performance capture in real-time using a custom 3D depth sensor based on structured light. The substantial amount of training data necessary to build a statistical model for tracking was later reduced significantly using the example-based facial rigging algorithm developed by Li and collaborators~\cite{li2010example}. With consumer-level depth sensors becoming mainstream (e.g., Kinect, Realsense, etc.), a whole line of real-time facial animation research have adopted these dimension reduction techniques with focus on deployability. The work of~\cite{weise2011realtime} incorporated pre-recorded motion priors to ensure stable tracking for noisy depth maps present in commercially available 3D sensors, which resulted in the popular animation software, Faceshift~\cite{Faceshift:2014:FS}. By optimizing the identity and expression models of the subject during tracking, Li and coworkers~\cite{li2013realtime}, as well as Bouaziz and collaborators~\cite{Bouaziz:2013:OMR} introduced an instant face tracking solution that does not require any user-specific calibration before tracking. To enable fully uninterrupted facial tracking under severe occlusions, Hsieh and colleagues~\cite{hsieh2015unconstrained} recently proposed an explicit facial segmentation technique based on depth and RGB cues. While the objectives of generating segmented 3D face models are most similar to this work, their method relies on depth sensor input.
%
%While the generation of 3D facial animations from pure RGB input have been demonstrated using sparse 2D landmarks detection~\cite{Pighin:etal:1999,Chuang02Performance,Chai:2003:VCF}, a superior performance capture fidelity and robustness has only been demonstrated recently by Cao and coworkers~\cite{Cao:2013:SRR} using a 3D shape regression approach. Cao and colleagues~\cite{cao2014displaced} later extended the efficient two-level boosted regression technique introduced in~\cite{export:206636} to the 3D case in order to avoid user-specific calibration. Higher fidelity facial tracking from monocular video has also been demonstrated with additional high-resolution training data~\cite{cao2015real}, very large datasets of a person~\cite{totalmoving}, or more expensive non-real-time computation~\cite{garrido2013reconstructing,shi2014automatic}. While robust to unconstrained lighting environments and large head poses, these methods are sensitive to large occlusions and cannot easily segment facial regions as with depth sensors~\cite{hsieh2015unconstrained}. Consequently, occlusions caused by hairstyles and hand gestures must be avoided, and many facial manipulation tasks such as face replacement require fully visible faces~\cite{Dale:2011:VFR,GVRTPT14,thies2015real}.
%
%Due to the immense variation of facial appearances in uncontrolled lighting environments, it is extremely challenging to segment facial regions at the pixel level.  
%The hierarchical CNN-based parsing network of Luo and collaborators~\cite{luo2012hierarchical} generates masks of individual facial components such as eyes, nose, and mouth even in the presence of occlusions, but does not segment the facial region as a whole. Smith and coworkers~\cite{smith2013exemplar} use an example-based approach for facial region and component segmentation, but the method requires sufficient visibility of the face. Besides demanding computation, these two methods are susceptible to wrong segmentations when occlusions have similar colors as the face. By alternating between face mask prediction and landmark localization with deformable part models, Ghiasi and Fowlkes~\cite{ghiasi2015using} have recently demonstrated state of the art facial segmentation results on the Caltech Occluded Faces in the Wild (COFW) dataset~\cite{burgos2013robust} at the cost of expensive computation steps. While there is relatively little work on facial segmentation from pure RGB, occlusion handling is an active research thread for facial recognition, where the modeling of partial occlusion has been proposed to be incorporated directly during training~\cite{5206862}. Similar ideas have been explored for the detection of 2D landmarks within an AAM frameworks~\cite{Gross:2006:AAM}, but superior results were later shown using techniques based on discriminatively trained deformable parts model ~\cite{Ramanan:2012:FDP,ghiasi2014occlusion}. Highly efficient landmark detection has been recently demonstrated using cascade of regressors trained with occlusion data~\cite{burgos2013robust,YuLBM14}. 
%
%In contrast to these methods, that aim at detecting sparse 2D landmarks robustly in the presence of occlusions, we explicitly compute a pixel-level face segmentation in real-time using a deep learning approach, and incorporate the resulting binary mask as prior into our regression-based dense 3D face tracker. While state of the art semantic segmentation networks have demonstrated impressive performances on general objects (e.g., humans, cars, etc.)~\cite{long_shelhamer_fcn,chen2014semantic,zheng2015conditional,liu2015semantic} in challenging unconstrained images, the robust and real-time segmentation of detailed facial regions remains unexplored.
%
%
%Due to the immense variation of facial appearances in uncontrolled lighting environments, it is extremely challenging to segment facial regions at the pixel level. Luo et al.~\cite{luo2012hierarchical} propose a hierarchical CNN-based parsing network to generate high quality masks of facial parts such as eyes, nose, and mouth. Smith et al.~\cite{smith2013exemplar} produce facial component masks including skin by aligning example images. While their works can robustly detect distinctive facial parts, occlusions in less distinct region such as skin poses further challenges. Besides, because it involves expensive computation steps, real-time usage is not possible. More recently, semantic segmentation with deep convolutional neural networks (DCNN) has achieved state of the art performance~\cite{long_shelhamer_fcn,chen2014semantic,zheng2015conditional,liu2015semantic}. We also base our facial segmentation on DCNN. To prevent overfitting to the training data, we synthetically augment training data.
%
%Occlusion handling is an active research thread for facial recognition, where the modeling of partial occlusion has been proposed to be incorporated directly during training~\cite{5206862}. Similar ideas have been explored for the detection of 2D landmarks within an AAM frameworks~\cite{Gross:2006:AAM}, but superior results were later shown using techniques based on discriminative trained deformable parts model ~\cite{Ramanan:2012:FDP,ghiasi2014occlusion}. Highly efficient landmark detection has been recently demonstrated using cascade of regressors trained with occlusion data~\cite{burgos2013robust,YuLBM14}. By alternating between face mask prediction and landmark localization with deformable part models, Ghiasi and Fowlkes~\cite{ghiasi2015using} have recently demonstrated state of the art results on the Caltech Occluded Faces in the Wild (COFW) dataset~\cite{burgos2013robust} at the cost of expensive computation steps. In contrast to these methods, that aim at detecting sparse 2D landmarks robustly in the presence of occlusions, we explicitly compute a pixel-level face segmentation using a deep learning approach, and use the resulting binary mask as prior to 3D face regressor.



%While sparse 2D features have been used to drive the animation of 3D face models~\cite{Pighin:etal:1999,Chuang02Performance,Chai:2003:VCF}, they  are generally 

%Graphics and Vision classic face capture
%
%==== 
%
%Graphics: production + high end + offline
%
%[offline/high cost/depth sensor/calibration/high-quality/controlled/manual] <--- vfx/gaming industry
%
%
%
%
%Vision / Surveillance / Analytics / Face Recognition
%
%landmark tracking:
%
%[offline / low cost / rgb / calibration / 2D landmarks / uncontrolled]
%\_ used for performance tracking
%
%[real-time / low cost / rgb / calibration / 2D landmarks / uncontrolled] Data-driven AAM CLM <---- landmark 
%
%[real-time / low cost / rgb / no calibration / 2D landmarks / uncontrolled] landmark prediction supervised descent <--- landmark
%
%
%=====
%
%Modern application
%
%=====
%
%Low-Cost Deployable Production / Realtime Virtual Production
%
%realtime performance capture / dense tracking:
%
%[real-time/high cost/depth sensor/lot of calibration/med-quality/controlled] Face/Off Weise 2009  <--- landmark
%
%[less calibration] EBFR Li 2010
%
%[real-time/low cost/depth sensor/calibration/low-quality/controlled] Weise 2011 / Faceshift <--- landmark
%
%
% 
%[real-time/low cost/depth sensor/no calibration/low-quality/controlled] Bouaziz 2013
%
%[real-time/low cost/depth sensor/no calibration/med-quality/controlled] KMM Li 2013 <--- landmark
% 
%[real-time/low cost/rgb/calibration/low-quality/controlled] Cao 2013
%
%[real-time/low cost/rgb/no calibration/low-quality/robust]: Cao 2014  <--- landmark
%
%[real-time/low cost/rgb/calibration/high-quality/controlled]: Cao 2015, wrinkles as pre-processing
%
%=====
%
%Video Processing / High-quality tracking
%
%[offline/low cost/rgb/calibration/high-quality/shape from shading/controlled] Garrido 2013
%[offline/low cost/rgb/no calibration/high-quality/shape from shading/controlled] Shi 2014
%[offline/low cost/rgb/no calibration/high-quality/no-occlusion] Suwajanakorn et al, <--- landmark internet images ira ICCV 2015, ECCV 2014
%
%VR 
%
%[real-time/low cost/depth sensor/calibration/med-quality/controlled] Li 2015 <--- landmark
%
%
%
%
%AR / Consumers
%
%
%[real-time/low cost/depth sensor/no calibration/med-quality/uncontrolled,segmentation] FaceX Hsieh 2015 <--- landmark
%
%[real-time/low cost/rgb/no calibration/low-quality/uncontrolled,segmentation]: This work 2016
%
%
%
%Face Segmentation only 2D / Occlusion Handling
%
%face recognition with occlusions: jia and M in face recognition with occlusions
%2D facial feature tracking with occlusions: Gross et al. AAM
%state of the art based on distriminative trained deformable parts [34,23] Ramanan 2012, Ghiasi and Fowlkes 2014 
%cascade of regression trained with occlusion data [8] Burgos-Artizzu: Robust face landmark estimation under occlusion
%\_ COFW (precision recall and accuracy of landmark location)
%
%
%
%===== latest segmentation papers:
%
%ONLY LANDMARKS
%
%
%
%
%ONLY Landmarks, no segmentation (SENSITIVE TO STARTING POINT)
%
%State-of-the-art occlusion handling (separated visibility probability optimization from landmark estimation. unified framework for both self occlusion(profile view) and other occlusion)
%http://homepages.rpi.edu/~wuy9/ICCV15/FLD_iccv15.pdf
%\_ no segmentation, but landmarks, 
%
%
%SLOW
%
%
%HOG Feature and DPM (complicated and slow)
%
%Occlusion handling with segmentation (basically her CVPR paper with graph cut)
%http://www.ics.uci.edu/~fowlkes/papers/gf-faceocclusion-bmvc2015.pdf
%\_ with segmentation -> the best but very slow
%|
%based on the one below
%
%Deformable part model robust to occlusion (from the same author above). little a bit old school method and super slow (100x slower than caltech paper)
%http://www.ics.uci.edu/~gghiasi/papers/gf-cvpr14.pdf
%\_ local deformable part model
%
%
%
%EFFICIENT
%
%
%
%
%REGRESSION, part based and blend together, does not assume assume feature from a same area <--- our work is close to this
%
%A little extension of ICCV caltech paper, but using similar building blocks
%http://www.research.rutgers.edu/~xiangyu/paper/CoR_Occlusion_Robust_Face_Alignment.pdf
%\_ fast but not realtime
%
%
%
%Another segmentation paper, but not as good as the one from Ghiasi
%http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7084187&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel7%2F83%2F7086144%2F07084187.pdf%3Farnumber%3D7084187
%\_ coupled segmentation and regression -> better, just in 2D
%\_ ghiasi performs better
%\_ quality is shit
%
%
%OCCLUSION HANDLING
%
%ren et al. extract feature around landmark points, concatenate LBF based on regression <--- super fast because of LBF
%\_ coudl be more robust and accurate at least in 2D
%\_ linear projection 
%
%kazemi (dlib), intead of 2 level boosterd regressoin, they use gradient boost regression, using random forest instead of random fern,
%better feature extraction using better optimization, enfoce locality of features using probability, 
%
%both above are based on cao
%
%cao cvpr -> fast regression 2012 cvpr, face alignment by explicit shape regression <----- landmark
%\_ using random fern, pixel difference (RANDOM SAMPLE)
%
%
%
%
%
%
%
%LARGE POSES : face alignment across large poses: a 3D solution
%	\_ DNN

result.tex
\vspace{-0.4cm}
\section{Results}\label{sec:result}
\vspace{-0.1cm}
\begin{figure*}[t!]
 \centering
\includegraphics[width=\textwidth]{images/results/results}
\caption{ Results. We visualize the input frame, the estimated probability map, the facial segmentation over the tracked template, and the composited result. \vspace{-0.4cm}
%Large variance of input video that our system can robustly handle. Probability map can provide a rough estimation of face region but boundary may be inaccurate when there exists large occlusion. Face segmentation based on probability map can achieve higher result, in different kinds of occlusion. Face shape and transformation can be reliably recovered and mapped on the image.
 \label{fig:results}}
\end{figure*}

As shown in Figure~\ref{fig:results}, we demonstrate successful facial segmentation and tracking on a wide range of examples with a variety of complex occlusions, including hair, hands, headwear, and props. Our convolutional network effectively predicts a dense probability map revealing face regions even when they are blocked by objects with similar skin tones such as hands. In most cases, the boundaries of the visibile face regions are correctly estimated.  Even when only a small portion of the face is visibile we show that reliable 3D facial fitting is possible when processing input data with clean segmentations. In contrast to most RGB-D based solutions~\cite{hsieh2015unconstrained}, our method works seamlessly in outdoor environments and with any type of video sources. 



 
\paragraph{Segmentation Evaluation and Comparison.}



We evaluate the accuracy of our segmentation technique on 437 color test images from the Caltech Occluded Faces in the Wild (COFW) dataset~\cite{burgos2013robust}.
We use the commonly used intersection over union (IOU) metric between the predicted segmentations and the manually annotated ground truth masks provided by~\cite{jia2014structured}
in order to assess over and under-segmentations. We evaluate our proposed data augmentation strategy as well as the use of negative training samples in Figure~\ref{fig:deepeval} and
show that the explicit use of hand compositings significantly improves the probability map accuracy during hand occlusions. We evalute the architecture of our network in Table~\ref{tab:netcomp} (left) and Figure~\ref{fig:deepeval} and compared our results with the state-of-the-art out of the box segmentation networks, FCN-8s\cite{long_shelhamer_fcn}, DeconvNet~\cite{noh2015learning}, and the naive ensemble of DeconvNet and FCN (EDeconvNet). Compared to FCN-8s and Deconvnet, the IOU of our method is improved by $12.7 \%$ and $1.4 \%$ respectively, but also contains much less noise as shown in Figure~\ref{fig:deepeval}. While comparable to the performance of EDeconvNet, our method achieves nearly double the performance, which enables real-time capabilities (30 fps) on the latest GPU.

\begin{wrapfigure}{r}{0.45\textwidth}
\centering
\vspace{-0.85cm}
\includegraphics[width=0.43\textwidth]{images/results/deepeval}
\caption{Comparison of segmentation result based on different selection of neural network architectures.
 \label{fig:deepeval}}
\vspace{-0.5cm}
\end{wrapfigure}

We compare in Table~\ref{tab:netcomp} (right), our deep learning-based approach against the current state-of-the-art in facial segmentation: (1) the structured forest technique~\cite{yang2015robust}, (2) the regional predictive power method (RPP)~\cite{jia2014structured} and 3) segmentation-aware part model (SAPM)~\cite{ghiasi2014occlusion,ghiasi2015using}. We measure the IOU and two additional metrics: global (the percentage of all pixels that are correctly classified) and ave(face) (the average recall of face pixels), since the structured forest work~\cite{yang2015robust} uses these two metrics. We demonstrate superior performance to RPP (IOU: 0.833 vs 0.724) and structured forest (global: 0.882 vs 0.839, ave(face): 0.929 vs 0.886), and comparable result to SAPM (IOU: 0.833 vs 0.835, ave(face) 0.929 vs 0.871). Our method is significantly faster than SAPM which requires up to 30s per frame~\cite{ghiasi2014occlusion}. 

 
 \begin{table}[!htb]
    \begin{minipage}[t]{.40\linewidth}
      \vspace{0pt}
      \centering
        \begin{tabular}{ c | c | c }
            Network & mean IOU & FPS\\\hline\hline
            FCN-8s & 0.739 & 37.99\\\hline
            DeconvNet & 0.821 & 44.31\\\hline
            EDeconvNet  & 0.835 & 20.45\\\hline
            Our Method & 0.833 & 43.27\\
        \end{tabular}
        \captionsetup{labelformat=empty,labelsep=none}
    \end{minipage}
    \begin{minipage}[t]{.40\linewidth}
    \vspace{0pt}
      \centering
        \begin{tabular}{ c | c | c | c }
            Method & mean IOU & global & ave(face)\\\hline\hline
            Structured Forest~\cite{jia2014structured} & - & 0.839 & 0.886\\\hline
            RPP~\cite{yang2015robust} & 0.724  & - & - \\\hline
            SAPM~\cite{ghiasi2015using} & 0.835 & 0.886 & 0.871  \\\hline
            Our method & 0.833 & 0.882 & 0.929 \\\hline
            Our Method+GraphCut & 0.839 & 0.887 & 0.927 \\
        \end{tabular}
        \captionsetup{labelformat=empty,labelsep=none}
    \end{minipage} 
    \vspace{0.5cm}
    \caption{segmentation performance for different network structures (left) and state-of-the-art methods (right).}\label{tab:netcomp} \vspace{-0.6cm}
\end{table}

%<<<<<<< HEAD
%%Notice that the results of RPP and SAPM are obtained from over 507 test images of COFW including gray scale images, whereas ours doesn't include gray scale images.
%=======
%
% 
%\paragraph{Segmentation Evaluation and Comparison.}
%
%We evaluate the accuracy of our segmentation technique on 437 color test images from the Caltech Occluded Faces in the Wild (COFW) dataset~\cite{burgos2013robust}.
%We use the commonly used intersection over union (IOU) metric between the predicted segmentations and the manually annotated ground truth masks provided by~\cite{jia2014structured}
%in order to assess over and under-segmentations. We evaluate our proposed data augmentation strategy as well as the use of negative training samples in Figure~\ref{fig:deepeval} and
%show that the explicit use of hand compositings significantly improves the probability map accuracy during hand occlusions. We evalute the architecture of our network in Table~\ref{tab:netcomp} and Figure~\ref{fig:deepeval} and compared our results with the state-of-the-art out-of-the-box segmentation networks, FCN-8s\cite{long_shelhamer_fcn}, DeconvNet~\cite{noh2015learning}, and the naive ensemble of DeconvNet and FCN (EDeconvNet). Compared to FCN-8s and Deconvnet, the IOU of our method is improved by $12.7 \%$ and $1.4 \%$ respectively, but also contains much less noise as shown in Figure~\ref{fig:deepeval}. While comparable to the performance of EDeconvNet, our method achieves nearly double the performance, which enables real-time capabilities (30 fps) on the latest GPUs.
%
%\begin{wrapfigure}{r}{0.45\textwidth}
%\centering
%\vspace{-0.85cm}
%\includegraphics[width=0.43\textwidth]{images/results/deepeval}
%\caption{Comparison of segmentation result based on different selection of neural network architectures.
% \label{fig:deepeval}}
%\vspace{-0.5cm}
%\end{wrapfigure}
%
%We compare in Table~\ref{tab:netcomp}, our deep learning-based approach against the current state of the art in facial segmentation: (1) the structured forest technique~\cite{yang2015robust}, (2) the regional predictive power method (RPP)~\cite{jia2014structured} and 3) segmentation-aware part model (SAPM)~\cite{ghiasi2014occlusion,ghiasi2015using}. We measure the IOU and two additional metrics: global (the percentage of all pixels that are correctly classified) and ave(face) (the average recall of face pixels), since the structured forest work~\cite{yang2015robust} uses these two metrics. We demonstrate superior performance to RPP (IOU: 0.833 vs 0.724) and structured forest (global: 0.882 vs 0.839, ave(face): 0.929 vs 0.886), and comparable result to SAPM (IOU: 0.833 vs 0.835, ave(face) 0.929 vs 0.871). Our method is significantly faster than SAPM which requires up to 30 seconds per frame~\cite{ghiasi2014occlusion}. %Notice that the results of RPP and SAPM are obtained from over 507 test images of COFW including gray scale images, whereas ours doesn't include gray scale images.
%>>>>>>> 4d5c2538c0ae863944a576bf528f49428ddd6a2e

% \ramen{We evaluate the accuracy of our segmentation method on 437 color test images from the Caltech Occluded Faces in theWild (COFW) dataset~\cite{burgos2013robust} based on Intersection over Union (IOU) in the provided bounding box between predicted segmentations and the manually annotated mask provided by~\cite{jia2014structured}. To evaluate our segmentation network design, we first compare our segmentation network with FCN~\cite{long_shelhamer_fcn}, DeconvNet~\cite{noh2015learning}, and the naive ensemble of DeconvNet and FCN (denoted by EDeconvNet). As shown in Table~\ref{tab:netcomp}, our segmentation network improves 12.7 \% from FCN-8s and 1.4 \% from DeconvNet, and achieves comparable performance to EDeconvNet in half the time it takes with a single GPU setup. We also compare our method with state of the art facial mask reasoning methods: 1) structured forest ~\cite{yang2015robust}; 2) regional predictive power method (RPP)~\cite{jia2014structured}; 3) segmentation-aware part model (SAPM)~\cite{ghiasi2015using}.}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%
 % What to be written in the paper by Shunsuke
 %%%%%%%%%%%%%%%%%%%%%%%%%

% Figure~\ref{fig:results}
% successful facial segmentation and tracking on a wide range of examples with different type of occlusions, including hair, hands, headwear, and props.
% In contrast to most RGB-D based solutions~\cite{hsieh2015unconstrained}, our method works seamlessly in outdoor environments or any type of video sources. 
% Our convolutional network effectively predicts a dense probability map revealing face regions even when they are blocked by objects with similar skin tones such as hands. 

%==================================
% Segmentation comparison / evaluation
%==================================

 % we evaluate the accuracy of our segmentation method on 437 color test images from the Caltech Occluded Faces in theWild (COFW) dataset~\cite{burgos2013robust} based on Intersection over Union (IOU) in the provided bounding box between predicted segmentations and the manually annotated mask provided by~\cite{jia2014structured}.

% evaluation of our training strategy
 % Figure~\ref{fig:deepeval}
 % validate our training data augmentation
 % hand augmentation significantly improves the probability map

 % evaluation of our network choice
 % Table~\ref{tab:netcomp} and Figure~\ref{fig:deepeval}
 % compared with FCN-8s\cite{long_shelhamer_fcn}, DeconvNet~\cite{noh2015learning}, and the naive ensemble of DeconvNet and FCN (EDeconvNet)
 % compared to FCN-8s and DeconvNet, ours improves 12.7 \%, and 1.4 \% respectively
 % ours achieves comparable performance to EDeconvNet (0.833(ours) vs 0.835(EDeconvNet) in half the time EDeconvNet takes on a single GPU setup. 
 
 % comparison with other state of the art methods
 % Table~\ref{tab:netcomp}
% compared with 1) structured forest ~\cite{yang2015robust}; 2) regional predictive power method (RPP)~\cite{jia2014structured}}; 3) segmentation-aware part model (SAPM)~\cite{ghiasi2015using}
% we show IOU and two metrics: global (the percentage of all pixels that are correctly classified) and ave(face) (the average recall of face pixels) because Structured Forest paper use these two metrics.
% superior performance to RPP (IOU: 0.833 vs 0.724) and structured forest (global: 0.882 vs 0.839, ave(face): 0.929 vs 0.886), and comparable result to SAPM (IOU: 0.833 vs 0.835, ave(face) 0.929 vs 0.871)
% ours are significantly faster than SAPM which use deformable part models (their segmentation paper doesn't mention computational time, but their previous pape~\cite{ghiasi2014occlusion} which share the deformable part model as SAPM reports,~30 second per frame)
% note that the results of RPP and SAPM are over 507 test images of COFW including gray scale images, whereas ours doesn't include gray scale images)
% note that the results of structured forest are over a random subset of 300 images (out of 507) of COFW test data.

\paragraph{Tracking Evaluation and Comparison.}

%==================================
% Face tracking comparison / evaluation
%==================================

\begin{wrapfigure}{r}{0.45\textwidth}
 \centering
\includegraphics[width=0.43\textwidth]{images/results/robustness}
\caption{Challenging tracking scenes.
 \label{fig:robustness}}
\vspace{0.5cm}
\centering
\includegraphics[width=0.43\textwidth]{images/results/comparison.pdf}
\caption{Tracking comparison.
 \label{fig:comparison}}
\vspace{-0.5cm}
\end{wrapfigure}

In Figure~\ref{fig:robustness}, we highlight the robustness of our approach on extremely challenging cases. Our method can handle difficult lighting conditions, such as shadows and flashlights, as well as side views and facial hair. We further validate our data augmentation strategy during regression training and report quantitative comparisons with the current state-of-the-art method of Cao et al.~\cite{cao2014displaced} in Figure~\ref{fig:accuracy}. Here, we produce an unoccluded face as ground truth and synthetically generated occluding box with increasing size. 
In our experiment, we generated three sequences of 180 frames, covering a wide range of expressions, head rotations and translations. 
We observe that our explicit semantic segmentation approach is critical to ensuring high tracking accuracy. While using the masked training dataset for regression significantly improves robustness, we show that additional performance can be achieved by augmenting this data with additional synthetic occlusions. Figure~\ref{fig:comparison} shows how Cao et al.'s algorithm fails in the presence of large occlusions.  Our method shows comparable occlusion-handling capabilities as the work of~\cite{hsieh2015unconstrained} who rely an RGB-D sensor as input. We demonstrate superior performance to a recent robust 2D landmark estimation method~\cite{burgos2013robust} when comparing the projected landmark positions. 
In particular, our method can handle larger occlusions and head rotations.

%\begin{figure}[t!]
% \centering
%\includegraphics[width=\columnwidth]{images/results/accuracy}
%\caption{Error evaluation on different tracking methods. %The error of expression, rotation and translation are illustrated separately. Gray region in the background represent the fraction of region occluded, as shown on the right side Y axis. Note that a clear segmentation is required to guarantee better performance, and such a performance requires appropriate selection of network architecture as well as augmented training data.
% \label{fig:accuracy}}
%\end{figure}

% Show robustness of our approach on extremely challenging cases.
% Figure~\ref{fig:robustness}
% Our method can handle difficult lighting conditions, such as shadows and flashlights, as well as side views and facial hair.

% Quantitative comparison with state of the art + validation of our occlusion augmentation strategy in regression training
% Figure~\ref{fig:accuracy}
% our facial tracking framework using an unoccluded face as ground truth and synthetically generated occluding box with increasing size
% In our experiment, we generated three sequences of 180 frames for expression, rotation, and translations respectively
% our explicit semantic segmentation approach is critical to ensuring high tracking accuracy.
% additional performance can be achieved by augmenting this data with additional synthetic occlusions

% Qualitative comparison with state of the art
% Figure~\ref{fig:comparison}
% Cao et al.'s algorithm fails in the presence of large occlusions especially the color of occlusion is quite different from expected color in the training
% Our method shows comparable occlusion-handling capabilities as the work of~\cite{hsieh2015unconstrained} who use an RGB-D sensor as input.


%\paragraph{Tracking Comparison.}

%We compare the tracking accuracy with the current state of the art method of Cao et al.~\cite{cao2014displaced}. and our facial tracking framework without occlusion augmentation in Figure~\ref{fig:accuracy} using an unoccluded face as ground truth and synthetically generated occluding box with increasing size. 
%
%In our experiment, we generated three sequences of 180 frames, covering a wide range of expressions, head rotations and translations. We observe that our explicit semantic segmentation approach is critical to ensuring high tracking accuracy. While using the masked training dataset for regression significantly improves robustness, we show that additional performance can be achieved by augmenting this data with additional synthetic occlusions.
%
%
%As illustrated in Figure~\ref{fig:comparison}, Cao et al.'s algorithm fails in the presence of large occlusions. 
%Our method shows comparable occlusion-handling capabilities as the work of~\cite{hsieh2015unconstrained} who use an RGB-D sensor as input.

 %%%%%%%%%%%%%%%%%%%%%%%%%
 % Up to here
 %%%%%%%%%%%%%%%%%%%%%%%%%

%We also demonstrate superior performance of our method with a cutting edge algorithm for robust 2D landmark estimation under occlusions~\cite{burgos2013robust} when comparing the projected landmark positions. In particular, our method can handle larger occlusions and head rotations.

%
% We found under occlusion, [Cao et al. 14] would fail, while our method can keep accurate tracking.
% Comparing to RGB-D occlusion-handling method [Hsieh et al 14], our method show comparable performance with only RGB 
% input. We evaluate  2D landmark tracking performance under occlusion and we found our method is more robust than [Burgos-Artizzu et al. 
%13] under occlusion or large head rotation. (As a matter of fact, RGB-only method has advantage.... maybe this is for application or conclusion.)


 
%\begin{figure}[h!]
% \centering
%\includegraphics[width=\columnwidth]{images/results/comparison.pdf}
%\caption{Tracking comparison with the state of the art.
% \label{fig:comparison}}
%\end{figure}


%
%(1. a general view of performance)
%
%Our system can accurately and robustly capture 3D facial performance with only RGB input in realtime. In Figure 7, we show results 
%under various capture conditions. Like existing techniques, our method adapts to the facial expressions and identity of 
%a person during tracking, thus the system can capture detail performance in 
%uninterrupted (?) situation. Besides that, our method can handle occlusion of 
%different form and extent, regardless the variability of occlusion. We show that our system can produce accurate face mask and 
%3D facial shape, when part of face is occluded by hand, clothes or other common objects. 
%Our system can capture both live performance or recorded videos. We show results from internet videos where characters are occluded by microphone or other people.  
%We also show the system performs well under controlled or uncontrolled lighting 
%condition. In Figure 8 shows more results under extreme lighting conditions, special configuration of 
%face (profile view) or with facial hair. 
%
%(it's unconstrained lighting, faces with different skin colors, object with different appearances, fine details,
% very large occlusions, expressions are well resolved. we demonstrate robustness )
%
%(2. justify our system design)
%
%(2.1 segmentation network design)
%
%In Figure 7, the probability map from convolutional neural network shows clear contour of 
%visible face region. Combined with refinement process, the segmentation can 
%reveal more details of facial contour such as hair. The refinement could also potentially 
%correct the minor mis-estimation in the probability map. 
%
%In the segmentation network, we have multiple options for system structure. Figure 10 illustrates the probability map from convolution network in different 
%structure. In the first row we compare results between the two convolution net architecture. 
%Both trained on original dataset, the VGG can produce finer probability map than AlexNet. (Actually we're comparing FCN-VGG-8s with FCN-AlexNet-32s) The second row compares between two segmentation network structure. Both trained on 
%original dataset and adopting VGG architecture, the FCN can produce more plausible probability map compared to DeconvNet. 
%In our experiment, we found that the DeconvNet can predict finer details. However, with limited and unvaried (?) training data, it lacks the flexibility for different shape and locations of objects. 
%(The DeconvNet seems to assume the object appears in the center of image, since all original training data has face at center regions.)
%(Two points here: (1) VGG is more expressive than AlexNet; (2) with limited training data, FCN can do better.)
%
%In the last column Figure 10, we show that by training on augmented data (perturbation, random box and hand occlusion), 
%FCN+VGG can produce high quality probability map, which can handle various forms of occlusion. 
%
%
%(2.2 regression training)
%
%The importance of training with occluded data
%
%
% (2.3 system performance) 
%
%the quantitative analysis
% 
% We evaluate the tracking accuracy and robustness by synthesizing occlusion on input videos. 
% During evaluation, subject (the person) performs different motion (expression, face rotation and translation) 
% to test the tracking accuracy. The occlusion is a box rotating around the face center (?) and increasing size with time. 
% The error is computed by the RMS distance between tracked shape and ground truth 
% shape, which is obtained via our method on un-occluded input. 
% 
% We found that with data augmentation during regression training, our system performs better than that
% without augmentation. With VGG structure in segmentation network, the system is 
% more accurate than the AlexNet version. We found...

 



%like existing techniques, our method learns the facial expressions and identities of the person while tracking and we can switch between users.
%
%different levels of occlusions, synthetic datasets


%system evaluation:
%* segmentation performance:
%* different levels of training data (what does more add?)
%* comparison with alexnet vs VGG
%* comparison with FCN vs Deconvnet vs RamenNet
%* robustness with different parameters of part-based regression
%* if segmentation fails then part based segmentation can also fail despite its robustness.

%<<<<<<< HEAD
%=======
%\paragraph{Comparison.}
%
%In Figure 11, we compare capture performance with current state-of-the-art real-time facial performance capture methods.
% We found under occlusion, [Cao et al. 14] would fail, while our method can keep accurate tracking.
% Comparing to RGB-D occlusion-handling method [Hsieh et al 14], our method show comparable performance with only RGB 
% input. We evaluate  2D landmark tracking performance under occlusion and we found our method is more robust than [Burgos-Artizzu et al. 
%13] under occlusion or large head rotation. (As a matter of fact, RGB-only method has advantage.... maybe this is for application or conclusion.)
%
% 
%\begin{figure}[h!]
% \centering
%\includegraphics[width=\columnwidth]{images/results/comparison.pdf}
%\caption{Comparison with several state-of-the-art work on different applications. First row: work from Cao et al. is claimed to work under occlusion, however their method doesn't explicitly handle variant of occlusion and can fail with hands appearing on face. Second row: Hsieh et al. try to estimate the parameter of face using depth image. We show that similar result can be acquired without using depth sensor at all. Third row: Burgos-Artizzu et al. propose a robust facial landmark detector under occlusion, however for face with large occlusion it can still fail; also their work is limited to front view and doesn't handle head rotation very well, so it will also fail on side view without occlusion.
% \label{fig:comparison}}
%\end{figure}
%>>>>>>> 91119c60cb4036ecbe696a760f69e48e710c686b


%comparison with faceX
%* fails under daylight
%
%comparison with Cao et al.
%* fails under occlusions



%\paragraph{Limitations.} 
%
%* difficult segmentation cases
%* when it s very dark (large shadow on one side)
%* difficult expression tracking cases
%* if something is occluded motion is hard to resolve

\paragraph{Performance.}

Our tracking and segmentation stages run in parallel. The full facial tracking pipeline runs at 30 fps on a quad-core i7 2.8GHz Intel Core i7 with 16GB RAM and the segmentation is offloaded wirelessly to a quad-core i7 3.5GHz Intel Core i7 with 16GB RAM with an NVIDIA GTX Titan X GPU. During tracking, our system takes 18ms to regress the 3D face and 5ms to optimize the identity and the focal length. For segmentation, we measure the following timings: probability map computation 23ms, segmentation refinement 4ms, data transmission 1ms.  run on the GPU, and the remaining implementation is multi-threaded on the CPU.

%<<<<<<< HEAD
%\paragraph{Applications.}
%
%* face replacement and virtual make up
%* avatar animation in the wild
%
%=======
%\paragraph{Applications.}
%
%* face replacement and virtual make up
%* avatar animation in the wild
%
%>>>>>>> 91119c60cb4036ecbe696a760f69e48e710c686b
%\begin{figure}[t!]
% \centering
%\includegraphics[width=\columnwidth]{images/results/application}
%\caption{Our system has many applications
% \label{fig:application}}
%\end{figure}

%\begin{table}
%\begin{center}
%\caption{Comparison among different data augmentation methods}
%\begin{tabular}{| c | c | c | c | c |}
%\hline
%training data & IOU & global & ave(face)\\\hline
 
%O & 0.767 & 0.811 & 0.965\\\hline
%O+R & 0.822 & 0.872 & 0.938\\\hline
%O+R+H & 0.833 & 0.882 & 0.929\\\hline
%O+R+H+N & 0.811 & 0.873 & 0.877\\\hline

%\end{tabular}
%\end{center}
%\end{table}

segmentation.tex
\section{Facial Segmentation}\label{sec:probmap}
%\begin{figure}[h!]
% \centering
%\includegraphics[width=\columnwidth]{images/algorithm/segmentation}
%\caption{Segmentation pipeline. Input frame containing the full face is received by the segmentation neural network to provide a probability map of per-pixel likelihood to be in face region. Superpixel segmentation is applied also on the input frame and the probability map is used to guide a sharp and clear segmentation on it using graphcut algorithm.
% \label{fig:segmentation}}
%\end{figure}
%\vspace{-0.3cm}

Our facial segmentation pipeline computes a binary mask from the bounding box of a face in the input image. The cropped face image is first resized to a small $128\times128$ pixel resolution image, which is passed to a convolutional neural network for a dense 2-class segmentation problem. Similar to state-of-the-art segmentation networks~\cite{long_shelhamer_fcn,noh2015learning,chen2014semantic}, the overall network consists of two parts, (1) a lower convolution network for multi-dimensional feature extraction and (2) a higher deconvolution network for shape generation. This shape corresponds to the segmented object and is reconstructed using the features obtained from the convolution network.
The output is a dense $128\times128$ probability map that assigns each pixel to either a face or non-face region. While both state-of-the-art networks, FCN~\cite{long_shelhamer_fcn} and DeconvNet~\cite{noh2015learning} use the identical convolutional network based on VGG-16 layers~\cite{Simonyan14c}, they approach deconvolution differently. FCN performs a simple deconvolution using a single bilinear interpolation layer, and produces coarse, but clean overall shape segmentations, because the output layer is closely connected to the convolution layers preventing the loss of spatial information. DeconvNet on the other hand, mirrors the convolution process with multiple series of unpooling, deconvolution, and rectification layers, and generates detailed segmentations at the cost of increased noise. Noh and collaborators~\cite{noh2015learning} proposed to combine the outputs of both algorithms through averaging followed by a post-hoc segmentation refinement based on conditional random fields~\cite{NIPS2011_4296}, but the computation is prohibitively intensive. Instead, we develop an efficient network with shared convolution layers to reduce the number of parameters and operations, but split the deconvolution part into a two-stream architecture to benefit from the advantages of both networks. The output probability map resulting from a bilinear interpolation and mirrored deconvolution network are then concatenated before a final convolutional layer merges them into a single high-fidelity output map. We then use a standard graph cut algorithm~\cite{gridcut} to convert the probability map into a clean binary facial mask and upsample to the resolution of the original input image via bilinear interpolation.

%\ramen{Our segmentation pipeline takes an $128\times128$ image as input and provides a binary mask of face region. To solve this 2-class dense labeling problem, we adopt a deep convolutional neural network for semantic segmentation~\cite{long_shelhamer_fcn,noh2015learning,chen2014semantic}. Specifically we base our DNN on a state of the art segmentation network, DeconvNet~\cite{noh2015learning}. While a multi-layer deconvolution network makes DeconvNet capable of highly detailed segmentation, it is susceptible to mis-segmentation to similar objects due to its highly complex model. As reported in~\cite{noh2015learning}, another state of the art segmentation network, FCN, has complementary characteristics to DeconvNet; whereas FCN is not suitable to recover the detail of objects, it robustly detects the overall shape of an object because the output layer is closely connected to convolution layers without losing spacial information. Since face segmentation for facial tracking in real time requires high fidelity and robustness to arbitrary occlusions and motion blur, we propose a intrinsically fused DeconvNet with FCN-8s, called IF-DeconvNet, where both are sharing the same convolution layers. This allows our facial segmentation not only to be very robust and detailed, but also to enable real time performance because of its efficiency. In this section, we explain how our novel neural network IF-DeconvNet is trained with data augmentation and finetuning on negative mining dedicated to facial segmentation, and then describe how the resulting probability map is refined using a graphcut method. }



\paragraph{Architecture.}

\begin{wrapfigure}{r}{0.45\textwidth}
\vspace{-0.8cm}
\centering
\includegraphics[width=0.43\textwidth]{images/algorithm/architecture}
\caption{ConvNet architecture with two-stream deconvolution network.
 \label{fig:architecture}}
\vspace{-0.5cm}
\end{wrapfigure}

Our segmentation network consists of a single convolution network connected to two different deconvolution networks, DeconvNet and an 8 pixel stride FCN-8s as shown in Figure~\ref{fig:architecture}.
The network is based on a 16 layer VGG architecture and pre-trained on the PASCAL VOC 2012 data set with 20 object categories~\cite{Chatfield14}.
More specifically, VGG has 13 layers of convolutions and rectified linear units (ReLU), 5 max pooling layers, two fully connected layers, and one classification layer.
DeconvNet mirrors the convolutional network to generate a probability map with the same resolution as the input, by applying upsampling operations (deconvolution) and the inverse operation of pooling (unpooling). Even though deconvolution is fast, the runtime performance is blocked by the first fully connected layer which becomes the bottleneck of the segmentation pipeline. To enable real-time performance on a state-of-the-art GPU, we reduce the kernel size of the first fully connected layer from $7\times7$ to $4\times4$ pixels.

Further modifications to the FCN-8s are needed in order to connect the output of both DeconvNet and FCN deconvolution networks to the final convolutional layer. The output size of each deconvolution is controlled by zero padding, so that the size of each upsampled activation layer is aligned with the output of the previous pooling layer. While the original FCN uses the last fully connected layer as the coarsest prediction, we instead use the output of the last pooling layer, as the coarsest prediction in order to preserve spatial information like in DeconvNet. The obtained coarse prediction is then sequentially deconvoluted and fused with the output of pooling layer 4 and 3, and then a deconvolution layer upsamples the fused prediction to the input image size. Since our 2-class labeling problem is considerably less complex than multi-class ones, losing information from discarded layers would not really affect the segmentation accuracy. In the final layer, the output of both deconvolution networks are concatenated into a single matrix and we apply a $1\times1$ convolution to obtain a score map, followed by a softmax operation to produce the final fused probability map. In this way we can even learn blending weights between the two networks as convolution parameters, instead of a simple averaging of output maps as proposed by the separate treatment of~\cite{noh2015learning}. Please refer to the supplemental materials for the detailed configuration of our proposed network. 

%The outputs of two deconvolution networks are concatenated into a 4 channel matrix and applied to a $1\times1$ convolution layer. Thus, instead of simply averaging the outputs of both networks, we can also learn the blending weights for both networks as convolution parameters during the training process. 

%\ramen{
%Our segmentation net is composed of a single convolution part and two deconvolution networks: DeconvNet and FCN-8s, and the input of two deconvolution parts are connected to the convolution net. Following \cite{noh2015learning}, we adopt VGG 16-layer net pre-traind on the PASCAL VOC 2012 data with 20 object categories~\cite{Chatfield14} for the convolution operation. VGG has 13 layers of convolution and rectified linear unit (ReLU), 5 max pooling layers, and 2 fully connected layers. DeconvNet mirrors the convolutional processes to obtain probability map with the same resolution as the input by applying upsampling operation (deconvolution) and the inverse operation of pooling (unpooling). Although this mirrored upsampling scheme achieves the state of the art segmentation, the first fully connected layer becomes the computational bottleneck due to the large number of parameters to optimize, which prevents us from employing DeconvNet for realtime applications. We have addressed this performance problem by changing the kernel size of the first fully connected layer to $4\times4$ and it reduces the entire computation time by half. }

%\ramen{Connecting DeconvNet and FCN to the same convolution network, we make some modifications to FCN-8s. First, the output size of each deconvolution is controlled by zero padding so that the size of upsampled activation layer is aligned with the output of the previous pooling layer. Second, while the orignal FCN take the last fully connected layer as the coarsest prediction, we use the output of the last pooling layer as the prediction layer so that the prediction can preserve spacial information. Since our 2-class labeling problem is much less complex than multi-class semantic segmentation, losing information from the fully connected layers does not affect the segmentation accuracy. 
%
%
%The outputs of two deconvolution networks are concatenated into a 4 channel matrix and applied to a $1\times1$ convolution layer. 
%
%
%Thus, instead of simply averaging the outputs of both networks, we can also learn the blending weights for both networks as convolution parameters during the training process. Table~\ref{} shows the detailed configuration of our proposed network. 
%}

%\ramen{
%In the training, we initialize parameters with the pre-trained weights for the convolution network except the first fully connected layer because this kernel size of this layer is changed, and the first fully connected layers and deconvolution layers are initialized with zero-mean Gaussians. The loss function of IF-DeconvNet is the summation of softmax functions to the outputs of DeconvNet, FCN, and fused DeconvNet + FCN. 


%Note that we also test a single loss function computed by fused DeconvNet + FCN and find that only DeconvNet part is only trained and FCN part does not contribute to the segmentation. Thus we use multiple softmax loss functions from the outputs of the two networks and the fused layer. We set $0.01$, $0.9$, and $0.0005$ to the learning rate, momentum, and weight decay respectively. 


%}


\paragraph{Training.}

\begin{wrapfigure}{r}{0.45\textwidth}
\vspace{-0.8cm}
\centering
\includegraphics[width=0.43\textwidth]{images/algorithm/segtraining}
\caption{Segmentation training data.
 \label{fig:segtraining}}
\vspace{-0.5cm}
\end{wrapfigure}

%\begin{figure}[h!]
% \centering
%\includegraphics[width=\columnwidth]{images/algorithm/segtraining}
%\caption{Training data for segmentation
% \label{fig:segtraining}}
%\end{figure}

For an effective facial segmentation in unconstrained images, our convolutional neural network needs to be trained with large image datasets containing face samples and their corresponding ground truth binary masks. The faces should span a sufficiently wide range of shapes, appearance, and illumination conditions. We therefore collect $2927$ images from the LFW face database~\cite{LFWTech} and $5094$ images from the FaceWarehouse dataset~\cite{cao2014facewarehouse}. While the LFW dataset already contains pre-labeled face segmentations, we segment those in FaceWarehouse using a custom semi-automatic tool. We use the available fitted face templates to estimate skin tones and perform a segmentation refinement using a graph cut algorithm~\cite{Rother:2004:GIF}. Each sample is then manually inspected and corrected using additional seeds to ensure that occlusions such as hair and other accessories are properly handled. 

To prevent overfitting, we augment our dataset with additional $82,770$ images using random perturbations of translation, rotation, and scale. 
The data consist of mostly photographs with a large variety of faces in different head poses, expressions, and under different lightings. Occlusions through hair, hands, and other objects are typically avoided. We therefore generate additional $82,770$ samples based on random sized and uniformly colored rectangles on top of each face sample to increase the robustness to partial occlusions (see~\reffig{segtraining}). 

Skin toned objects such as hands and arms are commonly observed during hand-to-face gesticulations but are particularly challenging to segment due to similar colors as the face and fine structures such as fingers. We further augment the training dataset of our convolutional neural network with composited hands on top of the original $8021$ face images. We first captured and manually segmented 1092 hand images of different skin tones, as well as under different lighting conditions and poses. We then synthesized these hand images on top of the original face images, which yields  
$41380$ additional trainining samples using the same perturbation strategy. In total, $132,426$ images were generated to train our network. Our data-augmentation strategy can effectively train the segmentation network and avoid overfitting, even though only limited amount of ground truth data is available.

We initialize the training using pre-trained weights~\cite{Chatfield14} except for the first fully connected layer of the convolution network, since its kernel size is modified for our real-time purposes. Thus, the first fully connected layers and deconvolution layers are initialized with zero-mean Gaussians. The loss function is the sum of softmax functions applied to the output maps of DeconvNet, FCN, and their score maps.
The weights of each softmax function is set to $0.5$, $0.5$, and $1.0$ respectively, and the loss functions are minimized via stochastic gradient descent (SGD) with momentum for stable convergence. 
Notice that by only using the fused score map of DeconvNet and FCN for the loss function, only the DeconvNet model is trained and not FCN. We set $0.01$, $0.9$, and $0.0005$ as the learning rate, momentum, and weight decay, respectively. Our training takes 9 hours using 50,000 SGD iterations on a machine with 16GB RAM and NVIDIA GTX Titan X GPU.

We further fine-tune the trained segmentation by adding negative samples (containing no faces) based on hand, arm, and background images to a random subset of the training data so that the amount of negative samples is equivalent to positive ones. In particular, the public datasets contain images that are both indoor and outdoors. Similar techniques for negative data augmentation has been used previously to improve the accuracy of weak supervision-based classifiers~\cite{siva2012defence,song2014learning}. We use $4699$ hand images that contain no faces from the Oxford hand dataset~\cite{Mittal11}, and further perturb them with random translation and scalings. This fine-tuning with negative samples uses the same loss function and training parameters (momentum, weight decay, and loss weight) as with the training using positive data, but the initial learning rate is changed to 0.001. This training converges after $10,000$ SGD iterations and takes an additional $1.5$ hours of computation.

%Momentum, weight decay, loss functions, and loss weights for finetuning are the same as the first training and only initial learning rate is changed to 0.001. The finetune training converges after 10K iterations and takes 1.5 hour in the same environment for the first training.




%\ramen{
%We collected 2927 images from the LFW datasets~\cite{LFWTech} and 5094 images from the FaceWarehouse dataset~\cite{cao2014facewarehouse}. While the LFW dataset already contains face labeling, we manually labeled those in FaceWarehouse. To prevent overfitting, we augment our training data with random perturbations of translation, rotation, and scale. In addition, random-colored rectangles are placed on the faces to increase the robustness to partial occlusions (see~\reffig{segtraining}). Consequently, we generate 82,770 images through perturbations. We further improve the segmentation quality by finetuning the model trained with the augmented training data above. }


%\ramen{We augment the original 8021 face with hand images. We captured and manually segmented 1092 hand images under different lighting conditions, as well as different poses. We synthesized these hand images on top of the original face images, which results in 41380 additional trainining samples using the same perturbation strategy. In total, 132,426 images were generated to train our network. Our data augmentation strategy achieves efficient training from limited amount of ground truth data without overfitting.}


%We further improve the segmentation quality by augmenting our dataset with hand images, which share similar skin textures as the face. We captured and manually segmented 1092 hand images under different lighting conditions, as well as different poses. We synthesized these hand images on top of the original face images, which results in 41380 additional trainining samples using the same perturbation strategy. In total, 132,426 images were generated to train our FCN.

%\ramen{
%Skin-colored objects such as arm and hand are not only one of the most commonly observed occlusions on faces but also very challenging to segment out because similar texture to face confuses segmentation pipeline. Thus, we finetune the trained IF-DeonvNet by adding negative examples containing a hand or a arm to a random subset of the training data above so that the amount of negative examples is equivalent to positive examples. The technique of augmenting negative data for training has been widely used for classification problem with weak supervision to improve accuracy~\cite{siva2012defence,song2014learning}. Using 5628 hand images and bounding boxes provided by Oxford Hand Dataset~\cite{Mittal11}, we collect 4699 negative hand images in total by cropping images with random translation and scaling, and removing images including human faces. Momentum, weight decay, loss functions, and loss weights for finetuning are the same as the first training and only initial learning rate is changed to 0.001. The finetune training converges after 10K iterations and takes 1.5 hour in the same environment for the first training.
%}

\paragraph{Segmentation Refinement.}

%While our convolutional neural network can robustly estimate a facial probability map, if the input image is ambiguous due to motion blur or if face is severely overlapped with objects, the boundary of the probability map becomes less confident. Thus, we refine the segmentation mask using the probability map as unary term and pixel intensity for smoothness term in the fast grapch cut framework, GridCut~\cite{gridcut}. We apply graph-cut techniques to the same resolution as the resulting probability map $128\times128$ and bilinearly upsample the final segmentation result including alpha value to the original cropped image resolution. 

We convert the  $128\times128$ pixel probability map of the convolutional neural network to a binary mask using a standard graph cut algorithm~\cite{Rother:2004:GIF}.
Even though our facial segmentation is reliable and accurate, a graph cut-based segmentation refinement can purge minor artifacts such as small 'uncertainty' holes at boundaries, which can still appear for challenging cases such as (extreme occlusions, motion blur, etc.).
%, which also purges minor artifacts such as small 'uncertainty' holes which can still appear in the probability map. The  which is then bilinearly upsampled to its original input resolution. 
We optimize the following energy term between adjacent pixels $i$ and $j$ using the efficient GridCut~\cite{gridcut} implementation: 
\begin{equation}\label{equ:graphcut}
{\sum_i{\theta_i}(p_i) - \lambda \sum_{(i, j)} \theta_{i, j} }.
\end{equation}
The unary term $\theta_i(p_i)$ is determined by the facial probability map $p_i$, defined as $\theta_i(p_i) = -\log(p_i)$ for the sink and $\theta_i(p_i) = - \log(1.0 - p_i)$ for the source. The pairwise term $\theta_{i, j} = exp(- \frac{\lVert I_i-I_j \rVert^2}{2\sigma})$, where $I$ is the pixel intensity, $\lambda = 10$, and $\sigma = 5$.
The final binary mask is then bilinearly upsampled to the original cropped image resolution.


%Our segmentation pipeline takes an $128\times128$ image as input and provides a binary mask of face region. To solve this 2-class dense labeling problem, we adopt a deep convolutional neural network for semantic segmentation~\cite{long_shelhamer_fcn,chen2014semantic,noh2015learning} with the VGG network pre-traind on the PASCAL VOC 2012 data with 20 object categories~\cite{Chatfield14}. FCN is composed of convolution and deconvolution, which allows the coarse response map to recover the details. We initialize parameters with the pre-trained weights for the convolution network except the last fully connected layers. The fully connected layers and the deconvolution part are trained from scratch with a dedicated facial training dataset. Table~\ref{} shows the detailed configuration of our proposed network. 

%We fineturned the FCN with 2927 images from the LFW datasets~\cite{LFWTech} and 5094 images from the FaceWarehouse dataset~\cite{cao2014facewarehouse}. While the LFW dataset already contains face labeling, we manually labeled those in FaceWarehouse. To prevent overfitting, we augment our training data with random perturbations of translation, rotation, and scale. In addition, random-colored rectangles are placed on the faces to increase the robustness to partial occlusions (see~\reffig{segtraining}). Consequently, we generate 82,770 images through perturbations. We further improve the segmentation quality by augmenting our dataset with hand images, which share similar skin textures as the face. We captured and manually segmented 1092 hand images under different lighting conditions, as well as different poses. We synthesized these hand images on top of the original face images, which results in 41380 additional trainining samples using the same perturbation strategy. In total, 132,426 images were generated to train our FCN.

%The general problem of image segmentation among different categories of objects, including humans and cars, has been well studied using neural network architecture.
%
%
%Generally speaking, a neural network is a series of operations applied on a multi dimensional matrix for prediction.
%For instance, in our segmentation setting input is a 3 channel image, which can be generalized as a $3\times W\times H$ matrix, and the output is a $W\times H$ matrix indicating the face/non-face mask for each pixel.
%The training of neural network is the process to find the best parameters for each operation, by using large amount of data.
%For example, the parameter of a matrix multiplication is all the elements of a size defined matrix $M$, and the relevant operation is to multiply this matrix $M$ to the input vector and output a vector of another size.
%
%Normal operations such as matrix multiplication or 2-dimensional convolution, when connected together as a network, has shown to be powerful enough to outperform other machine learning methods under explicit assumption of models on signals.
%
%
%The key idea of neural network, which is a generalization of wavelet transform, is that with large enough amount of parameter and observations, we can always approximate the transformation happened to the input data, using a basis of transforms, without explicitly define the formulation itself.
%With dedicated designed architecture, including the operation and order of them, can achieve even higher accuracy and make what the state-of-the-art of image segmentation methods are.
%Lots of architectures are proven to be suitable for image classification, i.e. to get the label of the whole image.
%While such neural networks aim for extracting one single classification result of the whole image, segmentation neural network can be extended from classification neural network using standard approaches.
%Therefore, the choice of network architecture contains two folds: the choice of base network to extract information for every patch, and the choice of segmentation network on top to handle information between neighboring pixels.
%FCN, named after its novel fully convolution operation for extending classification network to segmentation network, is one of the state-of-the-art architecture suitable for this task.
%Other architecture such as exists as well, but we show that some of them are more computationally expensive or not suitable for face segmentation task. Thus we opt for the fully convolution layer to achieve the real-time performance.
%
%Specifically, the input data to FCN is a rough crop of original input image using a face location provided by the previous frame or face detection.
%This is to keep track on the consistent face across the video and reduce the amount of computation needed by the network.
%The output predicted by FCN is a same size image, where the value of each pixel is its probability to be within the face region.
%Note that results are computed per frame independently, and temporal filtering is not necessary to get a clear and consistent segmentation.
%
%While existing network can identify regions of different subjects, it's not yet proven to be suitable in distinguishing similar objects such as one's hands and face. A few modification are made, to obtain the result we show in the paper.

%While IF-DeconvNet can robustly estimate a facial probability map, the details of objects often appear smoothed out, since the deconvolution layer is closely connected to the output layer. Thus, we refine the segmentation mask, using the probability map as an unary term in graphcut. To further ensure realtime performance, we employ the GPU-based simple linear iterative clustering of (SLIC)~\cite{achanta_slic_2012} to generate \textit{superpixels} in the upsampled images. SLIC applies k-means clustering starting from $m \times m$ grid. Since SLIC ensures that the connectivity of the initial grid will not change, we convert superpixels into a $m \times m$ grid and perform graphcut so that the resulting segmentation minimizes the following scoring function:
%\begin{equation}\label{equ:graphcut}
%{\sum_i{\theta_i}(p_i) - \lambda \sum_{(i, j)} \theta_{i, j} }.
%\%end{equation}
%The unary term $\theta_i(p_i)$ is a sigmoid function of facial probability map $p_i$, defined as $\theta_i(p_i) = [1 - sig(\beta(p_i-\omega))]$ for sink and $\theta_i(p_i) = sig(\beta(p_i-%\omega))$ for source, where $\beta = 50$, and $\omega = 0.1$ for sink and $\omega = 0.7$ for source. The pairwise term $\theta_{i, j} = exp(-\lVert \I_i-\I_j \rVert^2)$, where $\I$ lies in% the CIELab color space and $\lambda = 1.2$.

%* need to explain why deep learning can do this??

%* briefly talk about deconvNet and FCN and difference between them. 

%* how deconvNet layer works -> at higher level, it's just upsampling 

%* FCN: apply convolution network for object classification while keeping the coarse spartial informaiton and upsample it by applying deconvolution.

%* say that our cropped image is aligned with the center of landmarks in the previous frame and adjust the size so that roughly we can have a same size of face. bacially the purpose of transtation, rotation, and scaling data augmentation is to conver possible range of face in the cropped image.

%* need to explain what's 8 pixel stride
%
%* 
 
system.tex
\section{Overview}\label{sec:system}
%1.Briefly describe what the whole system is, this is to prove our idea of segmentation
%2.put introduction of segmentation and tracking
%3.describe how we combine them

\begin{wrapfigure}{r}{0.45\textwidth}
\vspace{-0.8cm}
\centering
\includegraphics[width=0.43\textwidth]{images/overview/system.pdf}
\caption{Capture setting using a laptop integrated RGB webcam.\label{fig:system}}
\vspace{-0.5cm}
\end{wrapfigure}

\begin{figure*}[t!]
 \centering
\includegraphics[width=\textwidth]{images/overview/overview}
\caption{Overview of our facial segmentation and performance capture pipeline. 
 \label{fig:overview}}
\end{figure*}



%\begin{figure}[h!]
% \centering
%\includegraphics[width=\columnwidth]{images/overview/system.pdf}
%\caption{Capture Setting. Only ordinary web camera for RGB image capturing is needed. Also our capturing can still reliably work under large occlusion including presence of head mounted device.
% \label{fig:system}}
%\end{figure}


As illustrated in~\reffig{overview}, our system is divided into a facial segmentation stage (blue) and a performance capture stage (green).
Our pipeline takes an RGB image as input and produces a binary segmentation mask in addition to a tracked 3D face model, which is parameterized by a shape vector, as output.
The binary mask represents a per-pixel facial region estimated by a deep learning framework for facial segmentation. 
Following Cao et al.'s DDE regression technique~\cite{cao2014displaced}, the shape vector describes the rigid head motion and the facial expression coefficients, which drive the 
animation of a personalized 3D tracking model. In addition, the shape of the user's identity and the focal length are solved concurrently during performance capture.
While the resulting tracking model represents the 
shape of the subject, the shape vector can be used to retarget any digital character with compatible animation controls as input.

Our convolutional neural network first predicts a probability map on a cropped rectangular face region for which size and positions are determined based on the bounding box of the projected 3D tracking model from the previous frame. The face region of the initial frame is detected using the method of Viola and Jones~\cite{violajones}. The output probability map is a smaller fixed-size resolution 
image ($128\times 128$ pixels) and describes the likelihood for each pixel being labeled as part of the specific face region. While two output maps (one for the overall shape and one for fine-scaled details) are simultaneously produced by our two-stream deconvolution network, a single output probability map is generated through a final convolutional layer.
To ensure accurate and robust facial segmentation, we train our convolutional neural network using a large dataset of segmented face images, augmented with peturbations, synthetic occlusions, croppings, and hand compositings, as well as negative samples containing no faces. We convert the resulting probability map into a binary mask using a graph cut algorithm~\cite{gridcut} and bilinearly upsample the mask to the original input resolution. 
%, which also purges minor artifacts such as small 'uncertainty' holes which can still appear in the probability map. The  which is then bilinearly upsampled to its original input resolution. 

We then use this segmentation mask as input to the facial tracker as well as for compositing partial 3D facial models during occlusions. This facial segmentation technique is also used to produce training data for the regression model of the DDE framework. Our facial performance capture pipeline is based on the state-of-the-art method of~\cite{cao2014displaced}, which does not require any calibration step for individual users. The training process and the regression explicitly take the segmentation mask into account. Our system runs in real-time on commercially available desktop machines with sufficiently powerful GPU processors. For many mobile devices such as laptops, which are not yet ready for deep neural net computations, we can optionally 
offload the segmentation processing over Wi-Fi to a desktop machine with high-end GPU resources for real-time performance.

%
%Since our input data is small, we can make use of powerful machine equipped with modern GPUs for the neural network computation, and transmit probability map to other devices to enable performance capture on mobile (See \reffig{system}).





%We provide a system as illustrated in~\reffig{overview} to prove that the segmentation step can further improve the result of state-of-the-art algorithm in facial tracking.
%The input is a RGB video sequence with arbitrary size. By combining facial segmentation based on a  with a robust RGB facial tracking algorithm, our system estimates binary mask of facial visibility, 3D facial performance in realtime.
%
%Given a RGB image, initial estimation of facial segmentation is provided by a neural network architecture called fully convolutional network (FCN).
%The prediction of FCN will be further refined using ``superpixel and graphcut algorithm'', to produce a binary mask with face/non-face labeling for each pixel.
%
%% no idea about this -- cosimo
%For facial performance, we adopt the state-of-the-art regression-based face tracking method~\cite{cao2014displaced} by modifying its training process so that the regressor can explicitly incorporate dense facial visibility.



%Although FCN runs fast and is capable of distinguishing face from non-face object even if the texture of object is similar to facial parts (e.g., hand), the probability map may lack the detail of an object because of its structure simplicity. In addition, since it is very challenging to cover all variation of face (e.g., skin color, lighting, age, facial hair), FCN sometimes provides less decisive estimation on the boundary between two objects. Thus, we refine and segment facial area by making use of probability map and color information. More specifically, we apply graph cut with super-pixel to enhance scalability.  


%* facial segmentation
%* given an input RGB image, use a deep neural network to segment face region on a downsized image, we then obtain a probability map which is then refine at higher resolution to produce a per-pixel resolution segmentation in the form of a binary mask.
%* the segmentation with the convolutional network is the most intensive part which is performed on a high-end GPU. To enable real-time segmentation on mobile devices, we demonstrate the ability to offload this computation to a server via Wifi connection.

%* robust tracking
%* given an input image with a binary mask, the performance capture task consists of fitting a 3D model to a pre-segmented partial face.
%* the output is a face model that can be tracked without interruption and artifacts given occlusions, but also we can produce a pixel-level segmentation of the tracked model and demonstrate new capabilities such as face replacement and virtual make ups under occlusions due to hair and hands.

\input{segmentation}
\input{tracking}
tracking.tex

\vspace{-0.4cm}
\section{Facial Tracking}\label{sec:tracking}
\vspace{-0.1cm}
%\begin{figure}[h!]
% \centering
%\includegraphics[width=\columnwidth]{images/algorithm/regtraining}
%\caption{Training data for regression
% \label{fig:regtraining}}
%\end{figure}
%\section{Facial Tracking}\label{sec:tracking}

After facial segmentation, we capture the facial performance by regressing a 3D face model directly from the incoming RGB input frame. We adopt the state-of-the-art displaced dynamic expression (DDE) framework of~\cite{cao2014displaced} with the two-level boosted regression techniques of~\cite{export:206636} and incorporate our facial segmentation masks into the regression and training process. More concretely, instead of computing the regression on face images with backgrounds and occlusions, where appearance can take huge variations, we only focus on segmented face regions to reduce the dimensionality of the problem. While the original DDE technique is reasonably robust for sufficiently large training datasets, we show that processing accurately segmented images significantly improves robustness and accuracy, since only facial apperance and lighting variations need to be considered. Even skin toned occlusions such as hands can be handled effectively by our method. We briefly summarize the DDE-based 3D facial regression and then describe how to explicitly incorporate facial segmentation masks.

\paragraph{DDE Regression.} Our facial tracking is performed by regressing a facial shape displacement given the current input RGB image and an initial facial shape from the previous frame. Following the DDE model of~\cite{cao2014displaced}, we represent a facial shape as a linear 3D blendshape model, ($\b_0$, $\B$), with global rigid head motion $(\mathbf{R},\t)$ and 2D residual displacements $\D = [\d_1\ldots\d_m]^T \in \R^{2m}$ of $m=73$ facial landmark positions $\P = [\p_1\ldots\p_m]^T \in \R^{2m}$ (eye contours, mouth, etc.). 
We obtain $\P$ through perspective projection of the 3D face with 2D offsets $\D$:
\begin{equation}\label{equ:displace}
\p_i = \mathrm{\Pi}_f(\mathbf{R}\cdot(\mathbf{b}_0^i + \B^i\mathbf{x})+\t) + \d_i\quad,
\end{equation}
where $\b^i_0$ is the 3D vertex location corresponding to the landmark $\p_i$ in the neutral face $\b_0$, $\B = [\b_1, ..., \b_n]$ the bases of expression blendshapes, $\x \in [0, 1]^n$ the $n = 46$ blendshape coefficients based on FACS~\cite{ekmanfacial}. Each neutral face and expression blendshape is also represented by a linear combination of 50 PCA bases of human identity shapes~\cite{Blanz:1999:MMS} with $[\b_0, \B] = C_r \times \u$, $\u$ the user-specific identity coefficients, and $C_r$ the rank-3 core tensor obtained from the ZJU FaceWarehouse dataset~\cite{cao2014facewarehouse}. We adopt a pinhole camera model, where the projection operator $\mathrm{\Pi}_f: \R^3 \mapsto \R^2$ is specified by a focal length $f$. Thus, we can uniquely determine the 2D landmarks using the shape parameters $\S = \{\mathbf{R}, \t, \x, \D, \u, f\}$. 

While the goal of the regression is to compute all parameters $\S$ given an input frame $\I$, we separate the optimization of the identity coefficients $\u$ and the focal length $f$ from the rest, since they should be invariant over time. Therefore, the DDE regressor only updates the shape vector $\mathbf{Q} = [\mathbf{R}, \t, \x, \D]$ and $[\u,f]$ is computed only in specific key-frames and on a concurrent thread (see~\cite{cao2014displaced} for details). The two-level regressor structure consists of $T$ sequential cascade regressors $\{R_t(\I, \mathbf{Q}_{t})\}_{t=1}^T$ with updates $\delta\mathbf{Q}_{t+1}$ so that $\mathbf{Q}_{t+1} = \mathbf{Q}_t  + \delta \mathbf{Q}_{t+1}$. Each of the weak regressors $R_t$ classifies a set of randomly sampled feature points of $\I$ based on the corresponding pre-trained update vector $\delta\mathbf{Q}_{t+1}$. For each $t$, we sample new sets of 400 feature points via Gaussian distribution on the unit square. Notice that these points are represented as barycentric coordinates of a Delaunay triangulation of the mean of all 2D facial landmarks for improved robustness w.r.t. facial transformations. Each $R_t$ consists of second layer of $K$ primitive cascade regressors based on random ferns of size $F$ (binary decision tree of depth $F$). Each fern regresses a weaker shape parameter update from a feature vector of $F$ pixel intensity differences of feature point pairs from the $400$ samples. The indices of feature point pairs are specified during training by maximizing the correlation to the ground truth regression residuals. The training process also determines the random thresholds and bin classification values of each fern. 

At run-time, as described in~\cite{cao2014displaced}, if a new expression or head pose is observed, we collect the resulting shape parameters $\hat{\S}$ as well as the landmarks $\hat{\P}$, and alternate the updates of the identity coefficients $\u$ and the focal length $f$ by minimizing the offsets $\hat{\D}$ in \refequ{displace} for $L$ collected key-frames until it converges as follows:
\begin{equation}\label{equ:identity_op}
\argmin_{\u, f}{ \sum_{l=1}^L{\sum_{i=1}^m{\Vert \mathrm{\Pi}_f(\mathbf{\hat{R}}_l\cdot(\mathbf{b}_0^i(\u) + \B^i(\u)\cdot\mathbf{\hat{x}}_l)+\hat{\t}_l) - \hat{\p}_{l,i} \Vert^2}}}.
\end{equation}

% minimizes the residual to the ground truth $\mathbf{Q}^g$ among all training data $N$

\paragraph{Training.}

% \centering
%\includegraphics[width=\columnwidth]{images/algorithm/regtraining}
%\caption{Training data for regression
% \label{fig:regtraining}}
%\end{figure}

\begin{wrapfigure}{r}{0.45\textwidth}
\vspace{-0.8cm}
\centering
\includegraphics[width=0.43\textwidth]{images/algorithm/regtraining}
\caption{Regression training data.
 \label{fig:regtraining}}
\vspace{-0.5cm}
\end{wrapfigure}

The training process consists of constructing the ferns of the primitive regressors and specifying the $F$ pairs of feature point indices based on a large database of facial images with corresponding ground truth facial shape parameters. We construct the ground truth parameters $\{\S_i^g\}_{i=1}^M$ from a set of images $\{\I_i\}_{i=1}^M$ and landmarks $\{\P_i\}_{i=1}^M$. Given landmarks $\P$, the parameters of the ground truth $\S^g$ are computed by minimizing the following objective function $\Theta(\mathrm{R},\t,\x,\u,f)$:
\begin{equation}\label{equ:reconstruction}
\Theta(\mathrm{R},\t,\x,\u,f) = \sum_{i=1}^m{\Vert \mathrm{\Pi}_f(\mathbf{R}\cdot(\mathbf{b}_0^i(\u) + \B^i(\u)\cdot\mathbf{x})+\t) - \p_i \Vert^2}.
\end{equation}

As in~\cite{cao2014displaced}, we use $14,460$ labeled data from FaceWarehouse\cite{cao2014facewarehouse}, LFW\cite{LFWTech}, and GTAV\cite{tarres2012gtav} and learn a mapping from an initial estimation $\S^{*}$ to the ground-truth parameters $\S^g$ given an input frame $\I$. 
An initial set of $N$ shape parameters $\{\S^{*}_i\}_{i=1}^N$ are constructed by perturbing each training parameter in $\S$ within a predefined range. 
Let the suffix $g$ denote the ground-truth value, suffix $r$ a perturbed value. 
%Just like during runtime, we only update $\mathbf{Q} = [\mathbf{R}, \t, \x, \D]$ because $\u$ and $f$ are optimized separately.
%Since the identity $\u$ and the focal length $f$ are temporally invariant, while the regressors only update the concatenated vector $\mathbf{Q} = [\mathbf{R}, \t, \x, \D]$, $\u$ and $f$ are updated in a post-process. 

We construct the training dataset $\{\S^{*}_i = [\mathbf{Q}_i^r, \u_i^g, f_i^g], \S_i^g = [\mathbf{Q}_i^g, \u_i^g, f_i^g], \I_i\}_{i=1}^N$ and perturb the shape vectors with random rotations, translations, blendshape coefficients as well as, identity coefficients $\u^r$ and the focal length $f^r$ to improved robustness during training. Blendshapes are perturbed $15$ times and the other parameters $5$ times, resulting in a total of $506,100$ training data.
%for regression. In order to make regression more robust in case the identity $\u$ or the focal length $f$ is not accurate, the training set also includes perturbed $\u^r$ and $f^r$. 
%Note that since perturbed $\u^r$ and $f^r$ result in different projection of the 3D face, we adjust offsets $\D'_g$ so that $\S'^g = \{\mathbf{R}^g, \t^g, \x^g, \D'^g, \u^r, f^r\}$ provides the original ground-truth landmarks. 
The $T$ cascade regressors $\{R_t(\I, \mathbf{Q}_{t})\}_{t=1}^T$ then update $\mathbf{Q}$ so that the resulting vector $\mathbf{Q}_{t+1} = \mathbf{Q}_t  + \delta \mathbf{Q}_{t+1}$ minimizes the residual to the ground truth $\mathbf{Q}^g$ among all training data $N$. Thus the regressor at stage $t$ is trained as follows:
\begin{equation}\label{equ:fern}
\delta \mathbf{Q}_{t+1} = \argmin_{R}{ \sum_{i = 1}^{N}{ \lVert \mathbf{Q}^{g}_i  - ( \mathbf{Q}_{i,t} + R_t(\I, \mathbf{Q}_{i,t}) ) \rVert_2^2 }}.
\end{equation}

\paragraph{Optimization.} 
For both Equations~\ref{equ:identity_op} and~\ref{equ:reconstruction}, the blendshape and identity coefficients are solved using 3 iterations of non-linear least squares optimization with boundary constraints $\x \in [0, 1]^n$ using an L-BFGS-B solver~\cite{Byrd:1995:LMA} and the rigid motions $(\mathbf{R},\t)$ are obtained by interleaving iterative PnP optimization steps~\cite{Lu:2000:FGC}.

\paragraph{Segmentation-based Regression.}

%Enabling RGB-based realtime performance capture, we adopt state-of-the-art Displaced Dynamic Expression regressor (DDE)~\cite{cao2014displaced}. The regression-based tracking methods have demonstrated sufficient robustness regardless skin color, age, and lighting~\cite{cao2014displaced,cao20133d}. The presence of occlusion, however, poses a further challenge due to its immense space of appearance variability. Although regression-based approaches can handle occlusion commonly observed in a training dataset (e.g., sunglasses, beard) to some extent, it is not possible to predict all variety of occlusion in terms of shape and color due to its high dimensionality. In fact, when the color of occlusion is different from expected color, even the state of the art method may fail (See \reffig{comparison}). 

%Thus, we propose to reduce the dimension of inference space by explicitly incorporating the facial segmentation mask $\M$ in the regression processes. In this section, first we summarize DDE regression method and then describe how we incorporate the facial segmentation mask in DDE regression. 

%In DDE, the facial landmarks $\P = [\p_1; ...; \p_m]^T \in \R^{2m}$ are represented by projection of the 3D face and 2D offsets $\D = [\d_1; ...; \d_m]^T \in \R^{2m}$ as
%\begin{equation}\label{equ:displace}
%\p_i = \mathrm{\Pi}_f(\mathbf{R}\cdot(\mathbf{b}_0^i + \B^i\mathbf{x})+\t) + \d_i
%\end{equation}
%where $\b^i_0$ is the 3D vertex location corresponding to the landmark $\p_i$ in the neutral face $\b_0$, $\B = [\b_1, ..., \b_n]$ is the basis of expression blendshapes, $\x \in [0, 1]^n$ the $n = 46$ blendshape coefficients based on FACS~\cite{ekmanfacial}, and $(\mathbf{R},\t)$ the rigid motion of the head. The neutral face and blendshapes are also represented by a linear combination of 50 identity basis truncated by the N-mode SVD as $[\b_0, \B] = C_r \times \u$, where $\u$ is the user identity coefficients, and $C_r$ is the rank-3 core tensor from the facewarehouse dataset~\cite{cao2014facewarehouse}. Assuming a pinhole camera, the projection operator $\mathrm{\Pi}_f: \R^3 \mapsto \R^2$ is defined by a focal length $f$. Thus, the 2D landmarks are uniquely determined by the shape parameters $\S = \{\mathbf{R}, \t, \x, \D, \u, f\}$. 


%For training, we construct the ground truth parameters $\{\S_i^g\}_{i=1}^M$ from a set of images $\{\I_i\}_{i=1}^M$ and landmarks $\{\P_i\}_{i=1}^M$. Given landmarks $\P$, each parameter in the ground truth $\S^g$ are alternately computed by minimizing the following function $\Theta(\mathrm{R},\t,\x,\u,f)$:
%\begin{equation}\label{equ:reconstruction}
%\Theta(\mathrm{R},\t,\x,\u,f) = \sum_{i=1}^m{\Vert \mathrm{\Pi}_f(\mathbf{R}\cdot(\mathbf{b}_0^i(\u) + \B^i(\u)\cdot\mathbf{x})+\t) - \p_i \Vert^2}.
%\end{equation}
%As in~\cite{cao2014displaced}, we use a set of $m=73$ landmarks provided by~\cite{cao2014displaced} and the corresponding image from FaceWarehouse\cite{cao2014facewarehouse}, LFW\cite{LFWTech}, and GTAV\cite{tarres2012gtav}. 
%
%The goal of regression is to learn a mapping function from an initial estimation $\S^{*}$ to the ground-truth parameters $\S^g$ given an input frame $\I$. An initial set of $N$ shape parameters $\{\S^{*}_i\}_{i=1}^N$ are constructed by perturbing each training parameter in $\S$ within a predefined range. Let suffix $g$ denote the ground-truth value, suffix $r$ a perturbed value. Since the identity $\u$ and the focal length $f$ are temporally invariant, while the regressors only update the concatenated vector $\mathbf{Q} = [\mathbf{R}, \t, \x, \D]$, $\u$ and $f$ are updated in a post-process. Thus, we construct training set $\{\S^{*}_i = [\mathbf{Q}_i^r, \u_i^g, f_i^g], \S_i^g = [\mathbf{Q}_i^g, \u_i^g, f_i^g], \I_i\}_{i=1}^N$ for regression. In order to make regression more robust in case the identity $\u$ or the focal length $f$ is not accurate, the training set also includes perturbed $\u^r$ and $f^r$. 
%%Note that since perturbed $\u^r$ and $f^r$ result in different projection of the 3D face, we adjust offsets $\D'_g$ so that $\S'^g = \{\mathbf{R}^g, \t^g, \x^g, \D'^g, \u^r, f^r\}$ provides the original ground-truth landmarks. 
%Given the training set, $T$ regressors $\{R_t(\I, \mathbf{Q}_{t})\}_{t=1}^T$ in the cascade updates $\mathbf{Q}$ so that the resulting vector $\mathbf{Q}_{t+1} = \mathbf{Q}_t  + \delta \mathbf{Q}_{t+1}$ minimizes the residual to the ground truth $\mathbf{Q}^g$ among all training data $N$. Thus the regressor at stage $t$ is trained as follows:
%\begin{equation}\label{equ:fern}
%\delta \mathbf{Q}_{t+1} = \argmin_{R}{ \sum_{i = 1}^{N}{ \lVert \mathbf{Q}^{g}_i  - ( \mathbf{Q}_{i,t} + R_t(\I, \mathbf{Q}_{i,t}) ) \rVert_2^2 }}.
%\end{equation}

%The regressors consists of the cascade of $K$ primitive regressors based on random fern consisting of $F$ features and random thresholds. Each regressor $R_t(\I, \mathbf{Q})$ draws image features using a set of pixel-differences based on correlation between image features and residuals $\{\mathrm{\d}\mathbf{Q}_{i,t}\}_{i=1}^N$, where $\mathrm{\d}\mathbf{Q}_{i,t} = \mathbf{Q}^g_i - \mathbf{Q}_{i,t}$~\cite{cao2014face}. In order to achieve feature extraction invariant under large variation of pose and expression, the reference pixels of features are determined by the barycentric coordinates embedded on the triangulated facial landmarks. As in \cite{cao2014displaced}, in all of our experiments, we use $T = 15$, $K = 300$, and $F = 5$. At run-time, if a new expression or head pose is observed, we collect the resulting shape parameters $\hat{\S}$ and the landmarks $\hat{\P}$ and alternately update the identity coefficients $\u$ and the focal length $f$ by minimizing the offsets $\hat{\D}$ in \refequ{displace} among $L$ collected frames until it converges as follows:
%\begin{equation}\label{equ:identity_op}
%\argmin_{\u, f}{ \sum_{l=1}^L{\sum_{i=1}^m{\Vert \mathrm{\Pi}_f(\mathbf{\hat{R}}_l\cdot(\mathbf{b}_0^i(\u) + \B^i(\u)\cdot\mathbf{\hat{x}}_l)+\hat{\t}_l) - \hat{\p}_{l,i} \Vert^2}}}.
%\end{equation}

%\paragraph{Segmentation-enabled Regression.}

To incorporate the facial mask $\M$ obtained from~\refsec{probmap} into the regressors $R_t(\I, \P_{t}, \M)$, we simply mark non-face pixels in $\I$ for both training and inference and prevent the regressors to sample features in non-face region. To further enhance the tracking robustness under arbitrary occlusions, which is equivalent to incomplete views after the segmentation process, we augment the training data by randomly cropping out parts on the segmented face images (see \reffig{regtraining}).
For each of the $506,100$ training data sets, we include one additional cropped version with a rectangle centered randomly around the face region with Gaussian distribution and covering up to $80\%$ of the face bounding box in width and height. \reffig{accuracy} and accompanied video shows that this occlusion augmentation significantly improves the robustness under various occlusions after data augmentation.
eccv2016submission.bbl
\begin{thebibliography}{10}

\bibitem{weise2011realtime}
Weise, T., Bouaziz, S., Li, H., Pauly, M.:
\newblock Realtime performance-based facial animation.
\newblock In: ACM Transactions on Graphics (TOG). Volume~30., ACM (2011) ~77

\bibitem{Bouaziz:2013:OMR}
Bouaziz, S., Wang, Y., Pauly, M.:
\newblock Online modeling for realtime facial animation.
\newblock ACM Trans. Graph. \textbf{32}(4) (2013)  40:1--40:10

\bibitem{li2013realtime}
Li, H., Yu, J., Ye, Y., Bregler, C.:
\newblock Realtime facial animation with on-the-fly correctives.
\newblock ACM Trans. Graph. \textbf{32}(4) (2013) ~42

\bibitem{Cao:2013:SRR}
Cao, C., Weng, Y., Lin, S., Zhou, K.:
\newblock {3D} shape regression for real-time facial animation.
\newblock ACM Trans. Graph. \textbf{32}(4) (2013)  41:1--41:10

\bibitem{cao2014displaced}
Cao, C., Hou, Q., Zhou, K.:
\newblock Displaced dynamic expression regression for real-time facial tracking
  and animation.
\newblock ACM Transactions on Graphics (TOG) \textbf{33}(4) (2014) ~43

\bibitem{cao2015real}
Cao, C., Bradley, D., Zhou, K., Beeler, T.:
\newblock Real-time high-fidelity facial performance capture.
\newblock ACM Transactions on Graphics (TOG) \textbf{34}(4) (2015) ~46

\bibitem{hsieh2015unconstrained}
Hsieh, P.L., Ma, C., Yu, J., Li, H.:
\newblock Unconstrained realtime facial performance capture.
\newblock In: Proceedings of the IEEE Conference on Computer Vision and Pattern
  Recognition. (2015)  1675--1683

\bibitem{Faceshift:2014:FS}
Faceshift:
\newblock (2014) \url{http://www.faceshift.com/}.

\bibitem{Blanz:1999:MMS}
Blanz, V., Vetter, T.:
\newblock A morphable model for the synthesis of {3D} faces.
\newblock In: SIGGRAPH '99. (1999)  187--194

\bibitem{cao2014facewarehouse}
Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.:
\newblock Facewarehouse: a 3d facial expression database for visual computing.
\newblock Visualization and Computer Graphics, IEEE Transactions on
  \textbf{20}(3) (2014)  413--425

\bibitem{long_shelhamer_fcn}
Long, J., Shelhamer, E., Darrell, T.:
\newblock Fully convolutional networks for semantic segmentation.
\newblock CVPR (to appear) (November 2015)

\bibitem{noh2015learning}
Noh, H., Hong, S., Han, B.:
\newblock Learning deconvolution network for semantic segmentation.
\newblock In: Computer Vision (ICCV), 2015 IEEE International Conference on.
  (2015)

\bibitem{Chatfield14}
Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.:
\newblock Return of the devil in the details: Delving deep into convolutional
  nets.
\newblock In: British Machine Vision Conference. (2014)

\bibitem{Krizhevsky_imagenetclassification}
Krizhevsky, A., Sutskever, I., Hinton, G.E.:
\newblock Imagenet classification with deep convolutional neural networks.
\newblock In: Advances in Neural Information Processing Systems.  2012

\bibitem{Rother:2004:GIF}
Rother, C., Kolmogorov, V., Blake, A.:
\newblock "grabcut": Interactive foreground extraction using iterated graph
  cuts.
\newblock In: ACM SIGGRAPH 2004 Papers. SIGGRAPH '04, New York, NY, USA, ACM
  (2004)  309--314

\bibitem{LFWTech}
Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.:
\newblock Labeled faces in the wild: A database for studying face recognition
  in unconstrained environments.
\newblock Technical Report 07-49, University of Massachusetts, Amherst (October
  2007)

\bibitem{PighinLewisCourse06}
Pighin, F., Lewis, J.P.:
\newblock Performance-driven facial animation.
\newblock In: ACM SIGGRAPH 2006 Courses. SIGGRAPH '06 (2006)

\bibitem{Guenter:1998:MF}
Guenter, B., Grimm, C., Wood, D., Malvar, H., Pighin, F.:
\newblock Making faces.
\newblock In: SIGGRAPH '98. (1998)  55--66

\bibitem{zhang-siggraph2004-stfaces}
Zhang, L., Snavely, N., Curless, B., Seitz, S.M.:
\newblock Spacetime faces: High resolution capture for modeling and animation.
\newblock ACM Trans. Graph. \textbf{23}(3) (2004)  548--558

\bibitem{FurukawaP09}
Furukawa, Y., Ponce, J.:
\newblock Dense {3D} motion capture for human faces.
\newblock In: CVPR. (2009)  1674--1681

\bibitem{Li:2009:RSV}
Li, H., Adams, B., Guibas, L.J., Pauly, M.:
\newblock Robust single-view geometry and motion reconstruction.
\newblock ACM Trans. Graph. \textbf{28}(5) (2009)  175:1--175:10

\bibitem{Bee11}
Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C.,
  Sumner, R.W., Gross, M.:
\newblock High-quality passive facial performance capture using anchor frames.
\newblock ACM Trans. Graph. \textbf{30} (2011)  75:1--75:10

\bibitem{fyffe2011comprehensive}
Fyffe, G., Hawkins, T., Watts, C., Ma, W.C., Debevec, P.:
\newblock Comprehensive facial performance capture.
\newblock In: Computer Graphics Forum. Volume~30., Wiley Online Library (2011)
  425--434

\bibitem{Bhat:2013:HFF}
Bhat, K.S., Goldenthal, R., Ye, Y., Mallet, R., Koperwas, M.:
\newblock High fidelity facial animation capture and retargeting with contours.
\newblock In: SCA '13. (2013)  7--14

\bibitem{Fyffe:2014:DHF}
Fyffe, G., Jones, A., Alexander, O., Ichikari, R., Debevec, P.:
\newblock Driving high-resolution facial scans with video performance capture.
\newblock ACM Trans. Graph. \textbf{34}(1) (December 2014)  8:1--8:14

\bibitem{li93threed}
Li, H., Roivainen, P., Forcheimer, R.:
\newblock 3-d motion estimation in model-based facial image coding.
\newblock TPAMI \textbf{15}(6) (1993)

\bibitem{bregler1994surface}
Bregler, C., Omohundro, S.:
\newblock Surface learning with applications to lipreading.
\newblock Advances in neural information processing systems (1994)  43--43

\bibitem{Black:1995:TRR}
Black, M.J., Yacoob, Y.:
\newblock Tracking and recognizing rigid and non-rigid facial motions using
  local parametric models of image motion.
\newblock In: ICCV. (1995)  374--381

\bibitem{Essa:1996:MTI}
Essa, I., Basu, S., Darrell, T., Pentland, A.:
\newblock Modeling, tracking and interactive animation of faces and heads using
  input from video.
\newblock In: Proceedings of the Computer Animation. (1996)  68--79

\bibitem{Decarlo:2000:OFC}
Decarlo, D., Metaxas, D.:
\newblock Optical flow constraints on deformable models with applications to
  face tracking.
\newblock Int. J. Comput. Vision \textbf{38}(2) (2000)  99--127

\bibitem{cootes2001active}
Cootes, T.F., Edwards, G.J., Taylor, C.J.:
\newblock Active appearance models.
\newblock IEEE Transactions on Pattern Analysis \& Machine Intelligence (6)
  (2001)  681--685

\bibitem{Cristinacce:2008:AFL}
Cristinacce, D., Cootes, T.:
\newblock Automatic feature localisation with constrained local models.
\newblock Pattern Recogn. \textbf{41}(10) (2008)  3054--3067

\bibitem{Saragih:2011:DMF}
Saragih, J.M., Lucey, S., Cohn, J.F.:
\newblock Deformable model fitting by regularized landmark mean-shift.
\newblock Int. J. Comput. Vision \textbf{91}(2) (2011)  200--215

\bibitem{xiong2013supervised}
Xiong, X., De~la Torre, F.:
\newblock Supervised descent method and its applications to face alignment.
\newblock In: Computer Vision and Pattern Recognition (CVPR), 2013 IEEE
  Conference on, IEEE (2013)  532--539

\bibitem{export:206636}
Cao, X., Wei, Y., Wen, F., Sun, J.:
\newblock Face alignment by explicit shape regression.
\newblock International Journal of Computer Vision (2013)

\bibitem{kazemi2014one}
Kazemi, V., Sullivan, J.:
\newblock One millisecond face alignment with an ensemble of regression trees.
\newblock In: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE
  Conference on, IEEE (2014)  1867--1874

\bibitem{ren2014face}
Ren, S., Cao, X., Wei, Y., Sun, J.:
\newblock Face alignment at 3000 fps via regressing local binary features.
\newblock In: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE
  Conference on, IEEE (2014)  1685--1692

\bibitem{weise2009face}
Weise, T., Li, H., Van~Gool, L., Pauly, M.:
\newblock Face/off: Live facial puppetry.
\newblock In: Proceedings of the 2009 ACM SIGGRAPH/eurographics symposium on
  computer animation, ACM (2009)  7--16

\bibitem{li2010example}
Li, H., Weise, T., Pauly, M.:
\newblock Example-based facial rigging.
\newblock ACM Trans. Graph. \textbf{29}(4) (2010)  32:1--32:6

\bibitem{Pighin:etal:1999}
Pighin, F.H., Szeliski, R., Salesin, D.:
\newblock Resynthesizing facial animation through {3D} model-based tracking.
\newblock In: ICCV. (1999)  143--150

\bibitem{Chuang02Performance}
Chuang, E., Bregler, C.:
\newblock Performance driven facial animation using blendshape interpolation.
\newblock Technical report, Stanford University (2002)

\bibitem{Chai:2003:VCF}
Chai, J., Xiao, J., Hodgins, J.:
\newblock Vision-based control of {3D} facial animation.
\newblock In: SCA '03. (2003)  193--206

\bibitem{totalmoving}
Suwajanakorn, S., Kemelmacher-Shlizerman, I., Seitz, S.:
\newblock Total moving face reconstruction.
\newblock In: Computer Vision – ECCV 2014. Volume 8692.
\newblock Springer International Publishing (2014)  796--812

\bibitem{garrido2013reconstructing}
Garrido, P., Valgaerts, L., Wu, C., Theobalt, C.:
\newblock Reconstructing detailed dynamic face geometry from monocular video.
\newblock ACM Trans. Graph. \textbf{32}(6) (2013)  158

\bibitem{shi2014automatic}
Shi, F., Wu, H.T., Tong, X., Chai, J.:
\newblock Automatic acquisition of high-fidelity facial performances using
  monocular videos.
\newblock ACM Transactions on Graphics (TOG) \textbf{33}(6) (2014)  222

\bibitem{luo2012hierarchical}
Luo, P., Wang, X., Tang, X.:
\newblock Hierarchical face parsing via deep learning.
\newblock In: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE
  Conference on, IEEE (2012)  2480--2487

\bibitem{smith2013exemplar}
Smith, B., Zhang, L., Brandt, J., Lin, Z., Yang, J.:
\newblock Exemplar-based face parsing.
\newblock In: Proceedings of the IEEE Conference on Computer Vision and Pattern
  Recognition. (2013)  3484--3491

\bibitem{ghiasi2015using}
Ghiasi, G., Fowlkes, C.:
\newblock Using segmentation to predict the absence of occluded parts.
\newblock In: Proceedings of the British Machine Vision Conference (BMVC).
  (2015)  22.1--22.12

\bibitem{burgos2013robust}
Burgos-Artizzu, X.P., Perona, P., Doll{\'a}r, P.:
\newblock Robust face landmark estimation under occlusion.
\newblock In: Computer Vision (ICCV), 2013 IEEE International Conference on,
  IEEE (2013)  1513--1520

\bibitem{Gross:2006:AAM}
Gross, R., Matthews, I., Baker, S.:
\newblock Active appearance models with occlusion.
\newblock Image Vision Comput. \textbf{24}(6) (2006)  593--604

\bibitem{Ramanan:2012:FDP}
Ramanan, D.:
\newblock Face detection, pose estimation, and landmark localization in the
  wild.
\newblock In: CVPR. (2012)  2879--2886

\bibitem{ghiasi2014occlusion}
Ghiasi, G., Fowlkes, C.C.:
\newblock Occlusion coherence: Localizing occluded faces with a hierarchical
  deformable part model.
\newblock In: Computer Vision and Pattern Recognition (CVPR), 2014 IEEE
  Conference on, IEEE (2014)  1899--1906

\bibitem{YuLBM14}
Yu, X., Lin, Z., Brandt, J., Metaxas, D.N.:
\newblock Consensus of regression for occlusion-robust facial feature
  localization.
\newblock In: ECCV. (2014)  105--118

\bibitem{violajones}
Viola, P., Jones, M.:
\newblock Robust real-time face detection.
\newblock International Journal of Computer Vision \textbf{57}(2) (2004)
  137--154

\bibitem{gridcut}
:
\newblock Gridcut.
\newblock \url{http://www.gridcut.com/}

\bibitem{chen2014semantic}
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.:
\newblock Semantic image segmentation with deep convolutional nets and fully
  connected crfs.
\newblock arXiv preprint arXiv:1412.7062 (2014)

\bibitem{Simonyan14c}
Simonyan, K., Zisserman, A.:
\newblock Very deep convolutional networks for large-scale image recognition.
\newblock CoRR \textbf{abs/1409.1556} (2014)

\bibitem{NIPS2011_4296}
Kr\"{a}henb\"{u}hl, P., Koltun, V.:
\newblock Efficient inference in fully connected crfs with gaussian edge
  potentials.
\newblock In Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F.,
  Weinberger, K.Q., eds.: Advances in Neural Information Processing Systems 24.
\newblock Curran Associates, Inc. (2011)  109--117

\bibitem{siva2012defence}
Siva, P., Russell, C., Xiang, T.:
\newblock In defence of negative mining for annotating weakly labelled data.
\newblock In: Computer Vision--ECCV 2012.
\newblock Springer (2012)  594--608

\bibitem{song2014learning}
Song, H.O., Girshick, R., Jegelka, S., Mairal, J., Harchaoui, Z., Darrell, T.:
\newblock On learning to localize objects with minimal supervision.
\newblock arXiv preprint arXiv:1403.1024 (2014)

\bibitem{Mittal11}
Mittal, A., Zisserman, A., Torr, P.H.S.:
\newblock Hand detection using multiple proposals.
\newblock In: British Machine Vision Conference. (2011)

\bibitem{ekmanfacial}
Ekman, P., Friesen, W.:
\newblock Facial action coding system: a technique for the measurement of
  facial movement. 1978.
\newblock Consulting Psychologists, San Francisco

\bibitem{tarres2012gtav}
Tarr{\'e}s, F., Rama, A.:
\newblock Gtav face database.
\newblock GVAP, UPC (2012)

\bibitem{Byrd:1995:LMA}
Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.:
\newblock A limited memory algorithm for bound constrained optimization.
\newblock SIAM J. Sci. Comput. \textbf{16}(5) (September 1995)  1190--1208

\bibitem{Lu:2000:FGC}
Lu, C.P., Hager, G.D., Mjolsness, E.:
\newblock Fast and globally convergent pose estimation from video images.
\newblock IEEE Trans. Pattern Anal. Mach. Intell. \textbf{22}(6) (June 2000)
  610--622

\bibitem{jia2014structured}
Jia, X., Yang, H., Lin, A., Chan, K.P., Patras, I.:
\newblock Structured semi-supervised forest for facial landmarks localization
  with face mask reasoning.
\newblock In: Proc. Brit. Mach. Vis. Conf.(BMVA). (2014)

\bibitem{yang2015robust}
Yang, H., He, X., Jia, X., Patras, I.:
\newblock Robust face alignment under occlusion via regional predictive power
  estimation.
\newblock (2015)

\end{thebibliography}
