3D Face Reconstruction with Geometry Details from a Single Image
Luo~Jiang
Juyong~Zhang$^\dagger$
Bailin~Deng
~\IEEEmembership{Member
~IEEE
\IEEEauthorblockN{Hao Li\IEEEauthorrefmark{1
3D face reconstruction from a single image is a classical and challenging problem, with wide applications in many areas. Inspired by recent works in face animation from RGB-D or monocular video inputs, we develop a novel method for reconstructing 3D faces from unconstrained 2D images, using a coarse-to-fine optimization strategy. First, a smooth coarse 3D face is generated from an example-based bilinear face model, by aligning the projection of 3D face landmarks with 2D landmarks detected from the input image. Afterwards, using local corrective deformation fields, the coarse 3D face is refined using photometric consistency constraints, resulting in a medium face shape. Finally, a shape-from-shading method is applied on the medium face to recover fine geometric details. Our method outperforms state-of-the-art approaches in terms of accuracy and detail recovery, which is demonstrated in extensive experiments using real world models and publicly available datasets.
SingleImageReconstruction_v2-full.tex
\documentclass[journal]{IEEEtran}
%
% If IEEEtran.cls has not been installed into the LaTeX system files,
% manually specify the path to it like:
% \documentclass[journal]{../sty/IEEEtran}





% Some very useful LaTeX packages include:
% (uncomment the ones you want to load)


% *** MISC UTILITY PACKAGES ***
%
%\usepackage{ifpdf}
% Heiko Oberdiek's ifpdf.sty is very useful if you need conditional
% compilation based on whether the output is pdf or dvi.
% usage:
% \ifpdf
%   % pdf code
% \else
%   % dvi code
% \fi
% The latest version of ifpdf.sty can be obtained from:
% http://www.ctan.org/tex-archive/macros/latex/contrib/oberdiek/
% Also, note that IEEEtran.cls V1.7 and later provides a builtin
% \ifCLASSINFOpdf conditional that works the same way.
% When switching from latex to pdflatex and vice-versa, the compiler may
% have to be run twice to clear warning/error messages.






% *** CITATION PACKAGES ***
%
%\usepackage{cite}
% cite.sty was written by Donald Arseneau
% V1.6 and later of IEEEtran pre-defines the format of the cite.sty package
% \cite{} output to follow that of IEEE. Loading the cite package will
% result in citation numbers being automatically sorted and properly
% "compressed/ranged". e.g., [1], [9], [2], [7], [5], [6] without using
% cite.sty will become [1], [2], [5]--[7], [9] using cite.sty. cite.sty's
% \cite will automatically add leading space, if needed. Use cite.sty's
% noadjust option (cite.sty V3.8 and later) if you want to turn this off
% such as if a citation ever needs to be enclosed in parenthesis.
% cite.sty is already installed on most LaTeX systems. Be sure and use
% version 5.0 (2009-03-20) and later if using hyperref.sty.
% The latest version can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/contrib/cite/
% The documentation is contained in the cite.sty file itself.






% *** GRAPHICS RELATED PACKAGES ***
%
\ifCLASSINFOpdf
  %\usepackage[pdftex]{graphicx}
  % declare the path(s) where your graphic files are
  % \graphicspath{{../pdf/}{../jpeg/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  % \DeclareGraphicsExtensions{.pdf,.jpeg,.png}
\else
  % or other class option (dvipsone, dvipdf, if not using dvips). graphicx
  % will default to the driver specified in the system graphics.cfg if no
  % driver is specified.
  % \usepackage[dvips]{graphicx}
  % declare the path(s) where your graphic files are
  % \graphicspath{{../eps/}}
  % and their extensions so you won't have to specify these with
  % every instance of \includegraphics
  % \DeclareGraphicsExtensions{.eps}
\fi
% graphicx was written by David Carlisle and Sebastian Rahtz. It is
% required if you want graphics, photos, etc. graphicx.sty is already
% installed on most LaTeX systems. The latest version and documentation
% can be obtained at:
% http://www.ctan.org/tex-archive/macros/latex/required/graphics/
% Another good source of documentation is "Using Imported Graphics in
% LaTeX2e" by Keith Reckdahl which can be found at:
% http://www.ctan.org/tex-archive/info/epslatex/
%
% latex, and pdflatex in dvi mode, support graphics in encapsulated
% postscript (.eps) format. pdflatex in pdf mode supports graphics
% in .pdf, .jpeg, .png and .mps (metapost) formats. Users should ensure
% that all non-photo figures use a vector format (.eps, .pdf, .mps) and
% not a bitmapped formats (.jpeg, .png). IEEE frowns on bitmapped formats
% which can result in "jaggedy"/blurry rendering of lines and letters as
% well as large increases in file sizes.
%
% You can find documentation about the pdfTeX application at:
% http://www.tug.org/applications/pdftex



% correct bad hyphenation here
\hyphenation{op-tical net-works semi-conduc-tor}


\usepackage{amsmath}
%\usepackage{mathptmx}
\usepackage[pdftex]{graphicx}
\usepackage{times}
%\usepackage[justification=centering]{caption}
\usepackage{caption2}
%\usepackage[marginclue,footnote,silent,draft]{fixme}

\usepackage{amsmath}
%To restore IEEEtranâ€™s ability to automatically break within multiline equations
%\interdisplaylinepenalty=2500


\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage[mathscr]{euscript}

\usepackage{bm}
\usepackage{subfigure}
\usepackage{algorithm}
\usepackage{algorithmic}
\usepackage{multirow}
\usepackage{wrapfig}
\usepackage{booktabs}
\usepackage{threeparttable}
\usepackage[english]{babel}
\usepackage{microtype}


\newcommand{\bfA}{\mathbf{A}}
\newcommand{\bfC}{\mathbf{C}}
\newcommand{\bfE}{\mathbf{E}}
\newcommand{\bfe}{\mathbf{e}}
\newcommand{\bfF}{\mathbf{F}}
\newcommand{\bfK}{\mathbf{K}}
\newcommand{\bfT}{\mathbf{T}}
\newcommand{\bft}{\mathbf{t}}
\newcommand{\bfR}{\mathbf{R}}
\newcommand{\bfw}{\mathbf{w}}
\newcommand{\bfU}{\mathbf{U}}
\newcommand{\bfu}{\mathbf{u}}
\newcommand{\bfI}{\mathbf{I}}
\newcommand{\bfn}{\mathbf{n}}
\newcommand{\bfN}{\mathbf{N}}
\newcommand{\bfs}{\mathbf{s}}
\newcommand{\bfS}{\mathbf{S}}
\newcommand{\bfd}{\mathbf{d}}
\newcommand{\bfp}{\mathbf{p}}
\newcommand{\bfq}{\mathbf{q}}
\newcommand{\bfr}{\mathbf{r}}
\newcommand{\bfG}{\mathbf{G}}
\newcommand{\bfL}{\mathbf{L}}
\newcommand{\bfV}{\mathbf{V}}
\newcommand{\bfX}{\mathbf{X}}
\newcommand{\bfz}{\mathbf{z}}
\newcommand{\bflambda}{\mathbf{\lambda}}
\newcommand{\bfrho}{\mathbf{\rho}}
\newcommand{\bfdelta}{\mathbf{\delta}}
\newcommand{\projdirection}{\mathbf{Z}}

\newcommand{\bfQ}{\mathbf{Q}}
\newcommand{\bfD}{\mathbf{D}}
\newcommand{\bfM}{\mathbf{M}}
\newcommand{\bfP}{\mathbf{P}}
\newcommand{\bfa}{\mathbf{a}}
\newcommand{\bfb}{\mathbf{b}}
\newcommand{\bfv}{\mathbf{v}}
\newcommand{\bfH}{\mathbf{H}}
\newcommand{\bfomega}{\bm{\omega}}
\renewcommand{\algorithmicrequire}{ \textbf{Input:}}
\renewcommand{\algorithmicensure}{ \textbf{Output:}}
\newcommand{\argmin}{\operatornamewithlimits{arg\,min}}
\newcommand{\argmax}{\operatornamewithlimits{arg\,max}}
\newcommand{\tabincell}[2]{\begin{tabular}{@{}#1@{}}#2\end{tabular}}

\newcommand{\dist}{{\textrm{appr}}}
\newcommand{\tv}{{\textrm{tv}}}
\newcommand{\odt}{{\textrm{odt}}}
\newcommand{\reg}{{\textrm{reg}}}
\newcommand{\wDist}{\omega_{\textrm{d}}}
\newcommand{\wEdge}{\omega_{\textrm{e}}}
\newcommand{\wNormal}{\omega_{\textrm{n}}}
\newcommand{\wc}{\mathbf{c}}
\newcommand{\wexp}{\mathbf{w}_{\textrm{exp}}}
\newcommand{\wid}{\mathbf{w}_{\textrm{id}}}
\newcommand{\widc}[1]{w_{\textrm{id}}^{(#1)}}
\newcommand{\didc}[1]{\delta_{\textrm{id}}^{(#1)}}
\newcommand{\wexpc}[1]{w_{\textrm{exp}}^{(#1)}}
\newcommand{\dexpc}[1]{\delta_{\textrm{exp}}^{(#1)}}
\newcommand{\efit}{E_{\textrm{fit}}}
\newcommand{\eshade}{E_{\textrm{shading}}}
\newcommand{\proj}{\bm{\Pi}}
\newcommand{\weightfit}[1]{\gamma_{#1}}
\newcommand{\weightalbedo}[1]{\mu_{#1}}
\newcommand{\drefc}[1]{\delta_{\textrm{r}}^{(#1)}}
\newcommand{\weightsubspace}{\mu_2}
\newcommand{\mvme}{\operatorname{MVME}}
\newcommand{\mvsd}{\operatorname{MVSD}}
\newcommand{\me}{\operatorname{ME}}
\newcommand{\sd}{\operatorname{SD}}
\newcommand{\sphcoef}{\bm{\xi}}
\newcommand{\graphlap}{\mathbf{K}^{i}}
\newcommand{\faceregion}{\Omega}
\newcommand{\normalmap}[1]{\mathbf{n}'_{#1}}
\newcommand{\heightfieldnormal}[1]{\widehat{\mathbf{n}}_{#1}}
\newcommand{\egrad}{E_{\textrm{grad}}}
\newcommand{\eclose}{E_{\textrm{close}}}
\newcommand{\esmooth}{E_{\textrm{smooth}}}
\newcommand{\eint}{E_{\textrm{int}}}
\newcommand{\weightnormalmap}[1]{\omega_{#1}}
\newcommand{\errorsymb}{\varepsilon}
\newcommand{\ceres}{\textsc{Ceres}}

%\pdfminorversion=6

\newcommand{\facewarehouse}{\mbox{\textsc{FaceWarehouse}}}
\newcommand{\bfm}{\mbox{\textsc{BFM2009}}}

\newcommand{\paraheading}[1]{\textbf{{#1}.~}}

\newtheorem{thm}{Theorem}
\newtheorem{lem}{Lemma}
\newtheorem{prop}{Proposition}


\usepackage[bookmarks,backref=true,linkcolor=black]{hyperref}
\hypersetup{
  pdfauthor = {},
  pdftitle = {},
  pdfsubject = {},
  pdfkeywords = {},
  colorlinks=true,
  linkcolor= black,
  citecolor= black,
  pageanchor=true,
  urlcolor = black,
  plainpages = false,
  linktocpage
}
\usepackage{color}

\definecolor{turquoise}{cmyk}{0.65,0,0.1,0.1}
\definecolor{purple}{rgb}{0.65,0,0.65}
\definecolor{dark_green}{rgb}{0, 0.5, 0}
\definecolor{orange}{rgb}{0.8, 0.2, 0.2}


\newcommand{\juyong}[1]{{\color{purple}[Juyong: #1]}}
\newcommand{\bailin}[1]{{\color{magenta}[Bailin:  #1]}}
\newcommand{\jiangluo}[1]{{\color{turquoise}[Jiangluo: #1]}}
\newcommand{\revise}[1]{{\color{black}#1}}

%\newcommand{\ca}[1]{{\color{red}            {#1}}}
%\newcommand{\cb}[1]{{\color{blue}            {#1}}}


\begin{document}
%
% paper title
% Titles are generally capitalized except for words such as a, an, and, as,
% at, but, by, for, in, nor, of, on, or, the, to and up, which are usually
% not capitalized unless they are the first or last word of the title.
% Linebreaks \\ can be used within to get better formatting as desired.
% Do not put math or special symbols in the title.
\title{3D Face Reconstruction with Geometry Details from a Single Image}



% author names and affiliations
% transmag papers use the long conference author name format.

\author{Luo~Jiang,   Juyong~Zhang$^\dagger$, Bailin~Deng,~\IEEEmembership{Member,~IEEE,}  Hao Li, and~Ligang~Liu,~\IEEEmembership{Member,~IEEE,}% <-this % stops a space
\IEEEcompsocitemizethanks{\IEEEcompsocthanksitem L. Jiang, J. Zhang, H. Li, and L. Liu are with School of Mathematical Sciences,
	University of Science and Technology of China.\protect\\
% note need leading \protect in front of \\ to get a newline within \thanks as
% \\ is fragile and will error, could use \hfil\break instead.
\IEEEcompsocthanksitem B. Deng is with  School of Computer Science and Informatics, Cardiff University.}% <-this % stops an unwanted space
\thanks{$^\dagger$Corresponding author. Email: \href{mailto:juyong@ustc.edu.cn}{\texttt{juyong@ustc.edu.cn}}.}
}

%\author{\IEEEauthorblockN{Hao Li\IEEEauthorrefmark{1},
%Juyong Zhang\IEEEauthorrefmark{1}\thanks{Corresponding author: %Juyong Zhang (email: juyong@ustc.edu.cn).},
%Jianmin Zheng\IEEEauthorrefmark{2},
%Jianfei Cai\IEEEauthorrefmark{2}}
%\IEEEauthorblockA{\IEEEauthorrefmark{1}School of Mathematical %Sciences,
%University of Science and Technology of China}
%\IEEEauthorblockA{\IEEEauthorrefmark{2}School of Computer %Engineering,
%Nanyang Technological University}}



% The paper headers
%\markboth{~}{~}
\markboth{Submitted to IEEE Transactions on Image Processing}
{Jiang \MakeLowercase{\textit{et al.}}: 3D Face Reconstruction with Geometry Details from a Single Image}
% The only time the second header will appear is for the odd numbered pages
% after the title page when using the twoside option.
%
% *** Note that you probably will NOT want to include the author's ***
% *** name in the headers of peer review papers.                   ***
% You can use \ifCLASSOPTIONpeerreview for conditional compilation here if
% you desire.




% If you want to put a publisher's ID mark on the page you can do it like
% this:
%\IEEEpubid{0000--0000/00\$00.00~\copyright~2014 IEEE}
% Remember, if you use this you must call \IEEEpubidadjcol in the second
% column for its text to clear the IEEEpubid mark.



% use for special paper notices
%\IEEEspecialpapernotice{(Invited Paper)}


% for Transactions on Magnetics papers, we must declare the abstract and
% index terms PRIOR to the title within the \IEEEtitleabstractindextext
% IEEEtran command as these need to go into the title area created by
% \maketitle.
% As a general rule, do not put math, special symbols or citations
% in the abstract or keywords.
\IEEEtitleabstractindextext{%
\begin{abstract}
3D face reconstruction from a single image is a classical and challenging problem, with wide applications in many areas. Inspired by recent works in face animation from RGB-D or monocular video inputs, we develop a novel method for reconstructing 3D faces from unconstrained 2D images, using a coarse-to-fine optimization strategy. First, a smooth coarse 3D face is generated from an example-based bilinear face model,  by aligning the projection of 3D face landmarks with 2D landmarks detected from the input image. Afterwards, using local corrective deformation fields, the coarse 3D face is refined using photometric consistency constraints, resulting in a medium face shape. Finally, a shape-from-shading method is applied on the medium face to recover fine geometric details. Our method outperforms state-of-the-art approaches in terms of accuracy and detail recovery, which is demonstrated in extensive experiments using real world models and publicly available datasets.
\end{abstract}

% Note that keywords are not normally used for peerreview papers.
\begin{IEEEkeywords}
Tensor Model, Shape-from-shading, 3D Face Reconstruction.
\end{IEEEkeywords}}



% make the title area
\maketitle


% To allow for easy dual compilation without having to reenter the
% abstract/keywords data, the \IEEEtitleabstractindextext text will
% not be used in maketitle, but will appear (i.e., to be "transported")
% here as \IEEEdisplaynontitleabstractindextext when the compsoc
% or transmag modes are not selected <OR> if conference mode is selected
% - because all conference papers position the abstract like regular
% papers do.
\IEEEdisplaynontitleabstractindextext
% \IEEEdisplaynontitleabstractindextext has no effect when using
% compsoc or transmag under a non-conference mode.







% For peer review papers, you can put extra information on the cover
% page as needed:
% \ifCLASSOPTIONpeerreview
% \begin{center} \bfseries EDICS Category: 3-BBND \end{center}
% \fi
%
% For peerreview papers, this IEEEtran command inserts a page break and
% creates the second title. It will be ignored for other modes.
\IEEEpeerreviewmaketitle



\section{Introduction}
Reconstruction of 3D face models using 2D images is a fundamental problem in computer vision and graphics~\cite{Stylianou2009}, with various applications such as face recognition~\cite{BlanzV03,ZhuLYYL15} and animation~\cite{CaoHZ14,Thies_2016_CVPR}. However, this problem is particularly challenging, due to the loss of information during camera projection.
 
In the past, a number of methods have been proposed for face construction using a single image. Among them, example-based methods first build a low-dimensional parametric representation of 3D face models from an example set, and then fit the parametric model to the input 2D image. One of the most well-known examples is the 3D Morphable Model (3DMM) proposed by Blanz and Vetter~\cite{BlanzV99}, represented as linear combination of the example faces. 3DMM is a popular parametric face model due to its simplicity, and has been the foundation of other more sophisticated face reconstruction methods~\cite{ZhuLYYL15}. Another approach to single image reconstruction is to solve it as \emph{Shape-from-shading} (SFS)~\cite{Zhang99shapefrom}, a classical computer vision problem of 3D shape recovery from shading variation. For example, Kemelmacher-Shlizerman and Basri~\cite{Kemelmacher-ShlizermanB11} reconstruct the depth information from an input face image, by estimating its lighting and reflectance parameters using a reference face shape.

While these existing approaches are able to produce high-quality reconstruction from a single image, they also come with limitations. Although example-based methods are simple and efficient, they rely heavily on the dataset, and may produce unsatisfactory results when the target face is largely different from those in the example set; moreover, due to the limited degrees of freedom of the low-dimensional model, these methods often fail to reproduce fine geometric details (such as wrinkles) that are specific to the target face. SFS-based methods are able to capture the fine-scale facial details from the appearance of the input image; however, they require prior knowledge about the geometry or illumination to resolve the ambiguity of the reconstruction problem, and may become inaccurate when the input image does not satisfy the assumptions. 

\begin{figure}[t!] 
	\centering
	\includegraphics[width=\columnwidth]{teaser.pdf}
	\caption{3D face reconstruction from a single image. Given an input image (left), we reconstruct a 3D face with fine geometric details (right, top row). The input image can be used as texture for rendering the reconstructed face (right, bottom row).}
	\label{fig:teaser}
\end{figure}

\begin{figure*}[t] 
	\centering
	\includegraphics[width=\textwidth]{overview.pdf}
	\caption{An overview of our coarse-to-fine face reconstruction approach.}
	\label{fig:pipeline}
\end{figure*}

In this paper, we propose a novel coarse-to-fine method to reconstruct a high-quality 3D face model from a single image. Our method consists of three steps:
\begin{itemize}
\item First, we compute a coarse estimation of the target 3D face, by fitting an example-based parametric face model to the input image. Our parametric model is derived from \facewarehouse{}~\cite{CaoWZTZ14} and \revise{the Basel Face Model (\bfm)}~\cite{Paysan2009}, two 3D face datasets with large variation in expression and identity respectively. The resulting mesh model captures the overall shape of the target face. 
\item Afterwards, we enhance the coarse face model by applying smooth deformation that captures medium-scale facial features; we also estimate the lighting and reflectance parameters from the enhanced face model. 
\item Finally, the illumination parameters and the enhanced face model are utilized to compute a height-field face surface according to the shading variation of the input image. This final model faithfully captures the fine geometric details of the target face (see Fig.~\ref{fig:teaser}).
\end{itemize}



Our method builds upon the strength of the existing approaches mentioned above: the example-based coarse face enables more reliable estimation of illumination parameters, and improves the robustness of the final SFS step; the SFS-based final face model provides detailed geometric features, which are often not available from example-based approaches. Our method outperforms existing example-based and SFS methods in terms of reconstruction accuracy as well as geometric detail recovery, as shown by extensive experimental results using publicly available datasets.


%\bailin{Maybe mention some comparison with \cite{RothTL15} and \cite{roth2016adaptive}.}


%The rest of this paper is structured as follows. Sec.~\ref{sec:Related} reviews existing approaches for face reconstruction from a single image. Sec.~\ref{sec:Overview} provides an overview of our coarse-to-fine approach, followed by Secs.~\ref{sec:coarse}, \ref{sec:medium}, and \ref{sec:fine} that explain the three steps of our pipeline, respectively. Sec.~\ref{sec:experiments} presents and discusses our experimental results, and Sec.~\ref{sec:Conclusion} concludes the paper. 



%To reconstruct a 3D face model from a single unconstrained image is a fundamental problem in computer vision and graphics. As a lot of information gets loss during the camera projection process, it is an inverse under-constrained problem which makes this reconstruction problem very challenging. \bailin{We should improve the organization of this part to show the application of face reconstruction from image.}With the development of 3D printing technology, we would like to have our own high-quality 3D face models. Moreover, the personalized face models can be applied in face alignment, tracking, recognition and animation. Since 3D faces have richer discriminative information, they are more suitable for face recognition~\cite{BlanzV03} than 2D images. Even we can normalize the pose and expression of 3D face models to render front-view images to improve face recognition~\cite{Taigman_2014_CVPR, ZhuLYYL15}. And real-time facial reenactment of a monocular target video sequence needs to use personalized models to achieve deformation transfer~\cite{Thies_2016_CVPR}.
%
%There are some convenient approaches to reconstruct face models, such as using multi-view stereo images or digital scanners \bailin{We need references for these two types of techniques here. Also, what exactly is "digital scanner"? Does it mean laser scanner, or any 3D scanner?}. Compared with them, the way of using a single image is more difficult and important~\bailin{Why is single image reconstruction ``more important''?}. Back in 1999, the 3D Morphable Model (3DMM)~\cite{BlanzV99} was introduced to recover a smooth face shape from one photograph. Later a multilinear model was applied in~\cite{VlasicBPP05} to transfer facial expressions from monocular videos to drive a target face mesh. Both techniques construct a space of face models using a face database, and search within the space for a face shape that matches with the input image or video. As a result, the reconstructed model is heavily influenced by the database. For example, if the database consist of only smooth face shapes, then it is impossible to reconstruct a face model with fine geometric details such as wrinkles. D Different from this, \cite{Kemelmacher-ShlizermanB11} recovered faces by utilizing an input image as a guide to "mold" a single reference 3D face model. 
%Later, \cite{ZhuLYYL15} and~\cite{roth2016adaptive} used a personalized template based on 3DMM, instead of the single reference to recover faces. Furthermore, \cite{roth2016adaptive} used the shading methods to achieve adaptive face reconstruction with multiple unconstrained images. These methods depend heavily on the initial face shape. Even now, it is still a difficult problem to find a trusted face model as the initial template in the case of one photograph.
%
%
%
%In this work, we propose a coarse-to-fine method to reconstruct a high-quality 3D face model from a single image. As the increasing attention has been paid to face research area, there are some 3D face databases with more individuals and expressions, such as \facewarehouse{}~\cite{CaoWZTZ14}. We first build a bilinear face model by assembling all the meshes in \facewarehouse{} as a third order tensor. Then we obtain a smooth coarse face model by means of aligning several 3D landmarks to 2D landmarks. Because the database contains a limited variety of facial geometries, the bilinear model represents a sub-space of face shapes and can not describe the person-specific features effectively. These features often reflect the global facial idiosyncrasies, and we apply additional global corrective deformation fields to enhance the bilinear model. In the mid-level, we first establish the correspondences between image pixels and triangles of the face mesh to estimate the shading and reflectance of the face image. Then we obtain a medium face model by optimizing the corrections such that the appearance of the improved model is as close as possible to the input image. The medium face model can be used to update the correspondences and shading cues. By iterating this process several times, the correspondences between 2D image and 3D model and the 3D model are optimized together. In the fine-level, we estimate the accurate photometric normals according to the improved model and updated shading cues. Then we use the surface-from-dense-normals algorithms to recover a high-quality detailed face shape. Fig.~\ref{fig:teaser} shows a detailed 3D face shape reconstructed by our proposed approach. 
%
%In summary, the contributions of this work include:
%\begin{itemize}
%\item We propose a coarse-to-fine approach that can provide a high-quality initial face model, which is used to recover a detailed face model from an unconstrained face image. Previous work used a single reference or a coarse model that is not good enough for one picture. 
%
%\item We enhance the bilinear model by embedding itself and additional global correction deformation fields into our optimization system based on the rendering equation.
%
%\item We estimate the photometric normals by minimizing the shading gradient errors, which is more robust than directly comparing the observed and computed irradiance values.
%\end{itemize}



\section{Related Work}
\label{sec:Related}

\paraheading{Low-dimensional models}
Human faces have similar global characteristics, for example the location of main facial features such as eyes, nose and mouth. From a perception perspective, it has been shown that a face can be characterized using a limited number of parameters~\cite{Sirovich1987,Meytlis2007}. The low dimensionality of the face space allows for effective parametric face representations that are derived from a collection of sample faces, reducing the reconstruction problem into searching within the parameter space. A well-known example of such representations is the 3DMM proposed in~\cite{BlanzV99}, which has been used for various face processing tasks such as reconstruction\revise{~\cite{BlanzV99, romdhani2005estimating, keller20073d, bas2016fitting, huber2016multiresolution}}, recognition~\cite{BlanzV03,ZhuLYYL15}, face exchange in images~\cite{Blanz2004-Exchange}, and makeup suggestion~\cite{Scherbaum2011}. Low-dimensional representations have also been used for dynamic face  processing. To transfer facial performance between individuals in different videos, Vlasic et al.~\cite{VlasicBPP05} develop a multilinear face model representation that separately parameterizes different face attributes such as identity, expression, and viseme. In the computer graphics industry, facial animation is often achieved using linear models called blendshapes, where individual facial expressions are combined to create realistic facial movements~\cite{Lewis2014-Blendshape}. The simplicity and efficiency of blendshapes models enable real-time facial animation driven by facial performance captured from RGBD cameras~\cite{Weise2009,WeiseBLP11,BouazizWP13,li2013realtime,HsiehMYL15} and monocular videos~\cite{CaoWLZ13,CaoHZ14,CaoBZB15,Thies_2016_CVPR}. When using low-dimensional face representations derived from example face shapes, the example dataset has strong influence on the resulting face models. For instance, it would be difficult to reconstruct a facial expression that deviates significantly from the sample facial expressions. In the past, during the development of face recognition algorithms, various face databases have been collected and made publicly available~\cite{Gross2005}. Among them, \revise{\bfm{}} provides 3DMM representation for a large variety of facial identities. Recently, Cao et al.~\cite{CaoWZTZ14} introduced \facewarehouse{}, a 3D facial expression database that provides the facial geometry of 150 subjects, covering a wide range of ages and ethnic backgrounds. Our coarse face modeling method adopts a bilinear face model that encodes identity and expression attributes in a way similar to~\cite{VlasicBPP05}. We use \facewarehouse{} and \revise{\bfm{}} as the example dataset, due to the variety of facial expressions and identities that they provide respectively.

\paraheading{Shape-from-shading}
Shape-from-shading (SFS)~\cite{Zhang99shapefrom,Durou2008} is a computer vision technique that recovers 3D shapes from their shading variation in 2D images. Given the information about illumination, camera projection, and surface reflectance, SFS methods are able to recover fine geometric details that may not be available using low-dimensional models. On the other hand, SFS is an ill-posed problem with potentially ambiguous solutions~\cite{Prados2006}. Thus for face reconstruction, prior knowledge about facial geometry must be incorporated to achieve reliable results. For example, symmetry of human faces has been used by various authors to reduce the ambiguity of SFS results~\cite{Shimshoni2000,Zhao2000,Zhao2001}. Another approach is to solve the SFS problem within a human face space, using a low-dimensional face representation~\cite{Atick1996,Dovgard2004}. Other approaches improve the robustness of SFS by introducing an extra data source, such as a separate reference face~\cite{Kemelmacher-ShlizermanB11}, as well as coarse reconstructions using multiview stereo~\cite{WuWMT11,IchimBP15} or unconstrained photo collections~\cite{Kemelmacher-Shlizerman2011-InTheWild,RothTL15,roth2016adaptive}. We adopt a similar approach which builds an initial estimation of the face shape and augment it with fine geometric details using SFS. Our initial face estimation combines coarse reconstruction in a low-dimensional face space with refinement of medium-scale geometric features, providing a more accurate initial shape for subsequent SFS processing. 

\section{Overview}
\label{sec:Overview}

This section provides an overview of our coarse-to-fine approach to reconstructing a high-quality 3D face model from a single photograph. Fig.~\ref{fig:pipeline} illustrates the pipeline of our method.

To create a coarse face model (Sec.~\ref{sec:coarse}), we first build a bilinear model from \facewarehouse{} and \revise{\bfm{}} to describe a plausible space of 3D faces; the coarse face shape is generated from the bilinear model by aligning the projection of its 3D landmarks with the 2D landmarks detected on the input image, using a fitting energy that jointly optimizes the shape parameters (e.g., identity, expression) and camera parameters. To further capture person-specific features that are not available from the bilinear model, we enhance the coarse face using an additional deformation field that corresponds to medium-scale geometric features (Sec.~\ref{sec:medium}); the deformation field is jointly optimized with the lighting and albedo parameters, such that the shading of the enhanced model is close to the input image. Afterwards, the resulting medium face model is augmented with fine geometric details (Sec.~\ref{sec:fine}): the normal field from the medium face model is modified according to the input image gradients as well as the illumination parameters derived previously, and the modified normal field is integrated to achieve the final face shape. 

%Table~\ref{tab:symbols} lists the main symbols that are used in the following sections.
%\begin{table}[!htb]
%\caption{Symbols Used In This Paper.}
%\label{tab:symbols}
%\centering
%\scalebox{1.0}{
%\begin{tabular}{ll}
%\hline
%Symbol &Meaning\\
%\hline  
%$\bfF$ &The face shape which is a $3 \times 11510$ matrix\\        
%$\bfU$ & 2D landmarks assembled as a $2 \times 68$ matrix\\       
%$\Pi_{Q}$ &An orthographic projection matrix\\
%$\bfR$ &A $3 \times 3$ rotation matrix \\
%$\bft$ &A $2 \times 1$ translation vector \\
%$\bfw_{c}$ &The camera parameters including $\Pi_{Q}$, $\bfR$, $\bft$\\
%$\bfw_{id}$ &A column vector of the identity weights \\
%$\bfw_{exp}$ &A column vector of the expression weights \\
%$\bfI_{i,j}$ &The gray-scale value at the pixel$\{i, j\}$ \\
%$\bfs_{i,j}$ &The illumination at the pixel $\{i, j\}$ \\
%$\bfn_{i,j}$, $\bfN$ &The normal at the pixel$\{i, j\}$, $\bfN$ stack all the $\bfn_{i,j}$ \\
%$\bfr_{i,j}$, $\bfr$ &The skin reflectance at the pixel$\{i, j\}$, $\bfr$ stack all the $\bfr_{i,j}$ \\
%$\bfz_{i,j}$, $\bfz$ &The depth value at the pixel$\{i, j\}$, $\bfz$ stack all the $\bfz_{i,j}$ \\
%$\bfG$ &The gradient matrix \\
%$\bfL$ &The Laplacian matrix \\
%\hline
%\end{tabular}}
%\end{table}

\section{Coarse Face Modeling}
\label{sec:coarse}
\paraheading{Preprocessing}
\revise{The \facewarehouse{} dataset contains head meshes of 150 individuals, each with 47 expressions. All expressions are represented as meshes with the same connectivity, each consisting of 11510 vertices. The \bfm{} dataset contains 200 face meshes, and each mesh consists of 53490 vertices. In order to combine the two datasets, \revise{we first mask the face region on the head mesh from \facewarehouse{} to extract a face mesh, and fill the holes in the regions of eyes and mouth, to obtain a simply connected face mesh consisting of 5334 vertices.} Afterwards, we randomly sample the parameter space for \bfm{} to generate 150 neutral face models, and deform the average face model from \facewarehouse{} to fit these models via nonrigid registration~\cite{sd2004}. Then we transfer the other 46 expressions of the \facewarehouse{} average face model to each of the 150 deformed face models based on the method in~\cite{sd2004}. In this way, we construct a new dataset containing 300 individuals (150 from \bfm{} and 150 from \facewarehouse{}), each with 47 expressions. We perform Procrustes alignment for all the face meshes in the dataset. Moreover, \bfm{} provides 199 principal components to span the surface albedo space, but these principal albedo components cannot be used for our new dataset directly due to different mesh connectivity. Thus we transfer their albedo information to the new mesh representation using the correspondence identified in the nonrigid registration, to construct 199 principal albedo components for our dataset. These principal components will be used in Sec~\ref{sec:medium}.}

\paraheading{Bilinear face model}
Following~\cite{VlasicBPP05}, we collect the vertex coordinates of \revise{all face meshes} into a third-order data tensor, and perform $2$-mode SVD reduction along the identity mode and the expression mode, to derive a bilinear face model that approximates the original data set. In detail, the bilinear face model is represented as a mesh with the same connectivity as those from the data set, and its vertex coordinates $\bfF \in \mathbb{R}^{3 \times N_v}$ are computed as
\begin{equation}\label{eq:dataset approximate}
\bfF=\bfC_{\textrm{r}} \times_{2} \wid^{T}\times_{3} \wexp^{T},
\end{equation}
where \revise{$N_v$ is the number of vertices}, $\bfC_{\textrm{r}}$ is the reduced core tensor computed from the SVD reduction, and $\wid{} \in \mathbb{R}^{100}, \wexp{} \in \mathbb{R}^{47}$ are column vectors for the identity weights and expression weights which control the face shape. Note that here we only reduce the dimension along the identity mode, in order to maintain the variety of facial expressions in the bilinear model. For more details on multilinear algebra, the reader is referred to~\cite{de1997signal}.

\begin{figure}[t] 
   \centering
      \includegraphics[width=0.85\columnwidth]{landmarks.pdf}
   \caption{Our coarse face reconstruction is based on aligning the projection of labeled 3D face landmarks (right) with 2D landmarks detected on the input image (left).}
  \label{fig:landmarks}
 \end{figure}

To construct a coarse face, we align 3D landmarks on the bilinear face model with corresponding 2D landmarks from the input image. First, we preprocess the bilinear face mesh to manually label 68 landmark vertices. Given an input image, we detect the face as well as its corresponding 68 landmarks using the method in~\cite{ChenRWCS14} (see Fig.~\ref{fig:landmarks} for an example). Assuming that the camera model is a weak perspective projection along the $Z$ direction, we can write the projection matrix as
$\proj=
\left[
\begin{matrix}
\alpha & 0 & 0 \\
0 & \alpha & 0 \\
\end{matrix}
\right]
$.
Then we can formulate the following fitting energy to align the projection of landmark vertices with the detected 2D landmarks
\begin{eqnarray}
\efit{} &=& \sum_{k=1}^{68} \left\| \proj{} \bfR \bfF_{v_k} + \bft - \bfU_{k} \right\|_2^2 \nonumber\\ 
&& + \: \weightfit{1}  \sum_{i=1}^{100} \left( \frac{\widc{i}}{\didc{i}}  \right)^2 + \weightfit{2} \sum_{j=1}^{47} \left( \frac{\wexpc{j}}{\dexpc{j}}  \right)^2. 
\label{eq:2D3DFitting}
\end{eqnarray}
Here $\bfF_{v_k} \in \mathbb{R}^{3}$ and $\bfU_{k} \in \mathbb{R}^{2}$ are the coordinates of the $k$-th 3D landmark vertex and the corresponding image landmark, respectively; translation vector $\bft \in \mathbb{R}^2$ and rotation matrix $\bfR \in \mathbb{R}^{3 \times 3}$ determine the position and pose of the face mesh with respect to the camera; $\widc{i}$ and $\wexpc{j}$ are components of weight vectors $\wid{}$ and $\wexp{}$, while $\didc{i}$ and $\dexpc{j}$ are the corresponding singular values obtained from the $2$-mode SVD reduction; $\weightfit{1}$ and $\weightfit{2}$ are positive weights. As in~\cite{BlanzV99}, the last two terms ensure parameters $\widc{i}$ and $\wexpc{j}$ have a reasonable range of variation. 
This fitting energy is minimized with respect to the shape parameters $\wid{}, \wexp{}$ and the camera parameters $\proj,\bfR,\bft$ via coordinate descent. 
%First, the shape parameters can be initialized as
%\begin{equation}\label{eq:initialization of shape parameters}
%\bfw_{id}=\frac{1}{50}\sum_{m=1}^{50}\bfw_{id}^{m}, \:\:\: \bfw_{exp}=\frac{1}{47}\sum_{n=1}^{47}\bfw_{exp}^{n},
%\end{equation}
%where $\bfw_{id}^{m}$ and $\bfw_{exp}^{n}$ mean the $m$-th identity weight and the $n$-th expression weight in the database respectively. 
First we fix the shape parameters and reduce the optimization problem to
\begin{equation}\label{eq:solving the camera parameters}
\min_{\proj,\bfR,\bft} ~~\sum_{k=1}^{68} \left\| \proj{} \bfR \bfF_{v_k} + \bft - \bfU_{k} \right\|_2^2,
\end{equation}
which is solved using the pose normalization method from ~\cite{Kemelmacher-Shlizerman2011-InTheWild}. 
Next we fix the camera and expression parameters, which turns the optimization into
\begin{equation}\label{eq:solving the shape parameters}
\min_{\wid{}} ~~ \sum_{k=1}^{68} \left\| \proj{} \bfR \bfF_{v_k} + \bft - \bfU_{k} \right\|_2^2 + \weightfit{1}  \sum_{i=1}^{100} \left( \frac{\widc{i}}{\didc{i}}  \right)^2.
\end{equation}
This is a linear least-squares problem and can be easily solved by solving a linear system. Finally, we fix the camera and identity parameters, and optimize the expression parameters in the same way as \revise{Eq.~\eqref{eq:solving the shape parameters}}. These steps are iteratively executed until convergence. In our experiments, four iterations are sufficient for convergence to a good result.
 
\begin{figure}[t] 
   \centering
      \includegraphics[width=0.75\columnwidth]{updatelandmarks.pdf}
   \caption{\revise{For a non-frontal face images (a), the labeled 3D face silhouette landmarks (shown in cyan in (d)) need to be updated for better correspondence with the detected 2D silhouette landmarks. We construct a set of horizontal lines connecting the mesh vertices (shown in cyan in (b) and (c)), and select among them a set of vertices representing the updated silhouette according to the current view direction (shown in red in (b) and (c)). The new 3D silhouette landmarks (shown in pink in (d)) are selected within the updated silhouette.}}
  \label{fig:updated_landmarks}
 \end{figure} 
\paraheading{Landmark vertex update}
The landmark vertices on the face mesh are labeled based on the frontal pose. For non-frontal face images, the detected 2D landmarks along the face silhouette may not correspond well with the landmark vertices (see Fig.~\ref{fig:updated_landmarks}(a) for an example). Thus after each camera parameter optimization step, we update the silhouette landmark vertices according to the rotation matrix $\bfR$, while keeping the internal landmark vertices (e.g., those around the eyes, the nose, and the mouth) unchanged. Similar to~\cite{CaoHZ14}, we preprocess the original face mesh to derive a dense set of horizontal lines that connect mesh vertices and cover the potential silhouette region from a rotated view (see Fig.~\ref{fig:updated_landmarks}(b) and~\ref{fig:updated_landmarks}(c)). Given a rotation matrix $\bfR$, we select from each horizontal line a vertex that lies on the silhouette, and project it onto the image plane according to the camera parameters $\proj,\bfR,\bft$. These projected vertices provide an estimate of the silhouette for the projected face mesh. Then for each 2D silhouette landmark, its corresponding landmark vertex is updated to the silhouette vertex whose projection is closest to it (see Fig.~\ref{fig:updated_landmarks}(d)).

To determine the silhouette vertex on a horizontal line, we select the vertex whose normal encloses the largest angle with the view direction. Since the face mesh is approximately spherical with its center close to the origin, we approximate the unit normal of a vertex on the rotated face mesh as $\frac{\bfR \bfv}{\|\bfR \bfv\|_2}$, where $\bfv$ is the original vertex coordinates. Then the silhouette vertex is the one with the smallest value of $\left|  \projdirection \cdot \frac{\bfR \bfv}{\|\bfR \bfv\|_2} \right|$ within the horizontal line, where $\projdirection = [0, 0, 1]^T$ is the view direction. 
\begin{figure}[htbp] 
   \centering
      \includegraphics[width=0.90\columnwidth]{differentpose.pdf}
   \caption{\revise{Silhouette update improves accuracy of the coarse face model. Each row shows an input image ((a) and (d)), the corresponding coarse face model with silhouette update ((b) and (e)), and the one without silhouette update ((c) and (f)). The updated silhouette is shown in red. The top row shows an example with $+30^{\circ}$ yaw, and the bottom row with $+45^{\circ}$ yaw.}}
  \label{fig:differentpose}
 \end{figure}
 
\revise{The silhouette update improves the accuracy of the coarse face model for non-frontal images, as shown in Fig.~\ref{fig:differentpose} for two examples with $+30^{\circ}$ and $+45^{\circ}$ yaws: without the silhouette update, the resulting model will become wider due to erroneous correspondence with between the detected landmarks and the silhouette landmarks. When the yaw becomes larger, the detected 2D landmarks become less reliable, and the coarse face model becomes less accurate even with silhouette update. Our approach does not work well for images with very large poses (beyond $60^{\circ}$ yaw) unless the invisible landmarks can be accurately detected. On the other hand, our pipeline can be combined with large-pose landmark detection algorithms to produce good results for such images. Some examples are shown in Fig.~\ref{fig:bigpose}.}


\section{Medium Face Modeling}
\label{sec:medium}

Although the coarse face model provides a good estimate of the overall shape, it may not capture some person-specific geometric details due to limited variation of the constructed data set (see Fig.~\ref{fig:enhance}). Thus we enhance the coarse face using smooth deformation that correspond to medium-scale geometric features, to improve the consistency between its shading and the input image. During this process we also estimate the lighting and the albedo. The enhanced face model and the lighting/albedo information will provide the prior knowledge required by the SFS reconstruction in the next section. In this paper, we convert color input images into grayscale ones for simplicity and efficiency. However, it is not difficult to extend the formulation to directly process color images.
 
\paraheading{Lighting and albedo estimation}
To compute shading for our face mesh, we need the information about lighting and surface reflectance. Assuming Lambertian reflectance, we can approximate the grayscale level $s_{i,j}$ at a pixel $(i,j)$ using second-order spherical harmonics~\cite{frolova2004accuracy}:
\begin{equation}
\label{eq:sphericalharmoniclighting}
s_{i,j} = r_{i,j} \cdot \max(\sphcoef^T \bfH(\bfn_{i,j}), 0).
\end{equation}
Here $r_{i,j}$ is the albedo at the pixel; $\bfn_{i,j}$ is the corresponding mesh normal, computed via
\begin{equation}
	\bfn_{i,j} = \frac{(\bfv_2^{i,j} - \bfv_1^{i,j}) \times (\bfv_3^{i,j} - \bfv_1^{i,j})}{\|(\bfv_2^{i,j} - \bfv_1^{i,j}) \times (\bfv_3^{i,j} - \bfv_1^{i,j})\|_2},
	\label{eq:meshnormal}
\end{equation}
where $\bfv_1^{i,j}, \bfv_2^{i,j}, \bfv_3^{i,j}$ are the vertex coordinates for the mesh triangle that corresponds to pixel $(i,j)$;
$\bfH$ is a vector of second-order spherical harmonics
\begin{equation}\label{eq:the second spherical harmonics}
 \bfH(\bfn) = [1, n_x, n_y, n_z, n_xn_y, n_xn_z, n_yn_z, n_x^2-n_y^2, 3n_z^2-1]^T, 
\end{equation}
and $\bm{\xi}$ is a vector of harmonics coefficients. For more robust estimation, we follow~\cite{BlanzV99} and parametrize the surface reflectance using a \emph{Principal Component Analysis} (PCA) model:
\begin{equation}
\label{eq:albedo pcd model}
r_{i,j} = \left(\bm{\Phi}_{0} + \sum_{l=1}^{N_r} w_{r}^{l}\bm{\Phi}_{l} \right) \cdot \mathbf{c}_{i,j},
\end{equation}
where $[c_{i,j}^{1}, c_{i,j}^{2}, c_{i,j}^{3}] \in \mathbb{R}^{3}$ is the barycentric coordinate of the triangle corresponding to $r_{i,j}$, $[\bm{\Phi}_0, \bm{\Phi}_1,...,\bm{\Phi}_{N_r}] \in \mathbb{R}^{N_v \times (N_r+1)}$ is a basis of vertex albedos with $N_v$ being the number of vertices of the face mesh, $\bfw_r=(w_{r}^{1},...,w_{r}^{N_r}) \in \mathbb{R}^{N_r}$ is a vector for the albedo weights; $\mathbf{c}_{i,j} \in \mathbb{R}^{N_v}$ is a vector whose components for the three vertices of the triangle that contains pixel $(i,j)$ are equal to the barycentric coordinates of the pixel within the triangle, and the components for other vertices are zero. \revise{Among the 199 principal albedo components derived from \bfm{}, we choose $N_r$ principal components with the largest variance as $\bm{\Phi}_1,...,\bm{\Phi}_{N_r}$. We set $N_r=100$ in our experiments.} The lighting and albedo are then estimated by solving an optimization problem
\begin{eqnarray}
\min_{\bfr, \bm{\xi}, \bfd} && \sum_{i,j} \left( r_{i,j}\sphcoef^T \bfH(\bfn_{i,j}) - I_{i,j} \right)^2 +\weightalbedo{1}\sum_{l = 1}^{N_r} \left\| \frac{w_{r}^{l}}{\drefc{l}} \right\|_2^2,
\label{eq:AlbedoLightingOptimization}
\end{eqnarray}
where vectors $\bfr, \bfd$ collect the values $\{r_{i,j}\}, \{d_{i,j}\}$, respectively; $I_{i,j}$ denotes the grayscale value at pixel $(i,j)$ of the input image; $\{\drefc{l}\}$ are the standard deviations corresponding to the principal directions; $\weightalbedo{1}$ is a user-specified positive weight. To optimize this problem, we first set $\bfw_r$ to zero and optimize the harmonics coefficients $\bm{\xi}$. Then we optimize the reflectance weights $\bfw_r$ while fixing $\bm{\xi}$. Both sub-problems reduce to solving a linear system. This process is iterated three times in our experiment.

\begin{figure}[t] 
   \centering
      \includegraphics[width=\columnwidth]{localbasis.pdf}
   \caption{Some Laplacian eigenfunctions of local regions on the face mesh (displayed via color coding).}
  \label{fig:corrections}
 \end{figure}

\paraheading{Facial detail enhancement}
With an estimate of lighting and albedo, we can now enhance the coarse face mesh to reduce the discrepancy between the mesh shading and the input image. We apply a smooth 3D deformation field to the $N_v$ vertices of the frontal face mesh to minimize the following discrepancy measure with respect to the vertex displacements $ \bfD \in \mathbb{R}^{3 \times N_v}$:
\begin{equation}
	\eshade{}(\bfD) = \sum_{i,j} \left( r_{i,j} \max(\sphcoef^T \bfH(\widetilde{\bfn}_{i,j}), 0)  - I_{i,j}\right)^2,
	\label{eq:deformationproblem}
\end{equation}
where $\{\widetilde{\bfn}_{i,j}\}$ are the new mesh face normals. \revise{Specifically, since our final goal is to recover a depth field defined on the facial pixels in the given image, we sum over the pixels in Eq.~\eqref{eq:deformationproblem}. The correspondence between pixels and triangles are computed by the Z-buffer method~\cite{strasser1974schnelle}.} However, this nonlinear least-squares problem can be very time-consuming to solve, due to the high resolution of the mesh. Therefore, we construct a low-dimensional subspace of smooth mesh deformations and solve the optimization problem within this subspace, which significantly reduces the number of variables. Specifically, if we measure the smoothness of a deformation field using the norm of its graph Laplacian with respect to the mesh, then the Laplacian eigenfunctions associated with small eigenvalues span a subspace of smooth deformations. Indeed, it is well known in 3D geometry processing that the Laplacian eigenvalues can be seen as the frequencies for the eigenfunctions, which indicate how rapidly each eigenfunction oscillates across the surface~\cite{Zhang2010-Spectral}. Thus by restricting the deformation to the subspace with small eigenvalues, we inhibit the enhancement of fine-scale geometric features, leaving them to the SFS reconstruction step in Sec~\ref{sec:fine}. Since most facial variations are local, we select some local regions on the mesh, and perform Laplacian eigenanalysis on each region separately (see Fig.~\ref{fig:corrections}). The selected eigenfunctions are then combined to span a space of facial variations.
\begin{figure*}[t] 
   \centering
      \includegraphics[width=\textwidth]{enhance.pdf}
   \caption{Quantitative results on the dataset~\cite{ValgaertsWBST12}. The input image and its ground truth shape are shown in the first column. In the other columns, we show different face reconstructions and their corresponding error maps (according to Eq.~\eqref{eq:ErrorMap}): the coarse face model, the medium face model, the fine reconstruction with and without medium face modeling, the coarse model with modified identity parameters, and the fine reconstruction with medium face modeling from the modified coarse face. In the bottom, we show the reconstruction error values.}
  \label{fig:enhance}
 \end{figure*}
Specifically, for the $i$-th selected region, we preprocess the frontal face mesh to construct its graph Laplacian matrix $ \graphlap \in \mathbb{R}^{N_v \times N_v}$ based on mesh connectivity, and add a large positive value to the $j$-th diagonal element if vertex $j$ is outside the selected region. Then we perform eigendecomposition to obtain $k + 1$ eigenvectors $\bfe_0^i, \bfe_1^i, \ldots, \bfe_k^i$ corresponding to the smallest eigenvalues $\lambda_0^i \leq \lambda_1^i \leq \ldots \leq \lambda_k^i$. Among them, $\bfe_0^i$ has a constant value inside the selected region, representing a translation of the whole region~\cite{Zhang2010-Spectral}. Since it does not represent variation within the region, we discard $\bfe_0^i$ to get $k$ eigenvectors $\bfE^i=[\bfe_1^i, \ldots, \bfe_k^i]$. Combing all the eigenvectors to span the $x$-, $y$-, and $z$-coordinates of the vertex displacement vectors, we represent the deformation field as
\begin{equation}
	\bfD = (\bfE \bm{\eta})^T,
	\label{eq:subspacedeformation}
\end{equation}
where $\bfE = [ \bfE^1, \ldots, \bfE^{N_e} ] \in \mathbb{R}^{N_v \times (k \cdot N_e)}$ stacks the basis vectors, and 
$\bm{\eta} = [ \lambda_1^1, \ldots, \lambda_k^1, \ldots, \lambda_1^{N_e}, \ldots, \lambda_k^{N_e} ]^T \in \mathbb{R}^{(k \cdot N_e) \times 3}$ collects their linear combination coefficients. Then the deformation is determined by solving the following optimization problem about $\bm{\eta}$:
\begin{equation}
	\min_{\bm{\eta}} ~~ \eshade{} (\bfD) + \weightsubspace \sum_{i=1}^{N_e} \sum_{j=1}^k \left\| \frac{\bm{\eta}_j^i}{\lambda_j^i} \right\|_2^2.
	\label{eq:ReducedDeformationOptimization}
\end{equation}

Here the second term prevents large deformations, with more penalty on basis vectors of lower frequencies; $\weightsubspace$ is a user-specified weight. Our formulation is designed to induce more enhancement for finer geometric features, since the coarse face already provides a good estimate of the overall shape. \revise{In our experiments, we set $k=5$ and $N_e=9$, which means we select nine local regions and the first five eigenfunctions of the corresponding Laplacian matrix for each region. These local regions are manually selected in a heuristic way. More specifically, given the mean face shape, we first compute the vertex displacements from its neutral expression to each of the other 46 expressions, and manually select nine regions with the largest variation as the local regions.}

As the number of variables are significantly reduced in \eqref{eq:ReducedDeformationOptimization}, this nonlinear least-squares problem can be solved efficiently using the Levenberg-Marquardt algorithm~\cite{Madsen2004}. We then apply the optimized deformation field to the frontal face mesh, and update the correspondence between image pixels and mesh triangles. With the new correspondences, we solve the optimization problems \eqref{eq:AlbedoLightingOptimization} and \eqref{eq:ReducedDeformationOptimization} again to further improve the lighting/albedo estimate and the face model. This process is iterated twice in our experiments.

\revise{Medium face modeling can improve the accuracy of medium-scale facial features such as those around the laugh lines, as shown in Figs.~\ref{fig:enhance} and Figs.~\ref{fig:basisfunction}. Fig.~\ref{fig:enhance} compares the fine face reconstruction results with and without medium face modeling. We can see that the use of medium face leads to more accurate results numerically and visually. Indeed, eigendeomposition of the Laplacian matrix corresponds to Fourier analysis of geometric signals defined on the mesh surface~\cite{Zhang2010-Spectral}, thus our use of basisvectors is similar to approximating the displacement from the coarse face to the ground truth shape in each local region using its Fourier components of lowest frequencies, which is a classical signal processing technique. On the other hand, our approach cannot reconstruct facial features whose frequency bands have limited overlap with those corresponding to the chosen basisvectors. One example is shown in Fig.~\ref{fig:basisfunction}, where the dimples cannot be reconstructed. Finally, as the medium face modeling is applied on local regions, it cannot reduce reconstruction errors of global scales. As an example, in Fig.~\ref{fig:enhance} we alter the identity parameters to generate a different coarse face model, and apply medium and fine face modeling. We can see that although medium and fine face modeling help to introduce more details, they cannot change the overall face shape.}
	
%	Fig.~\ref{fig:enhance} shows the reconstructed face model from each algorithm stages. We can observe that the deformation fields help to improve the reconstruction accuracy both numerically and visually. The laugh lines at the corners of the mouth of the medium face are more obvious than the coarse face in Fig.~\ref{fig:enhance}. The reconstruction improvement by the deformation fields can be also observed in Fig.~\ref{fig:basisfunction}. The coarse reconstruction can capture the global structure of face shape very well, but it is not good enough to describe common local shape variations (e.g., laugh lines), and cannot describe uncommon local shape variations (e.g., dimples) at all (see Fig.~\ref{fig:basisfunction}(a) and Fig.~\ref{fig:basisfunction}(b)). Our introduced deformation fields can help the coarse faces to have a more obvious description of common local variations (see Fig.~\ref{fig:basisfunction}(c)). Since our deformation fields contain only low, medium variations, we cannot recover uncommon local shape variations in the medium face modeling. And for face images with small shape variations (see Fig.~\ref{fig:basisfunction}(d)), the coarse face models could be good enough (see Fig.~\ref{fig:basisfunction}(e)), thus our deformation fields have a little help (see Fig.~\ref{fig:basisfunction}(f)).}

 \begin{figure}[t] 
   \centering
      \includegraphics[width=\columnwidth]{basisfunction.pdf}
   \caption{\revise{An input image with smile expression (a), and its coarse (b) and medium (c) face models. The use of Laplacian eigenvectors improves the accuracy of features around the laugh lines, but cannot reconstruct the dimples as the eigenvectors provide limited cover of their frequency band.}}
  \label{fig:basisfunction}
 \end{figure} 
\section{Fine Face Modeling}
\label{sec:fine}

As the final step in our pipeline, we reconstruct a face model with fine geometric details, represented as a height field surface over the face region $\faceregion$ of the input image. Using the medium face model and the lighting/albedo information computed in Sec.~\ref{sec:medium}, we first compute a refined normal map over $\faceregion$, to capture the details from the input image. This normal map is then integrated to recover a height field surface for the final face shape.

\paraheading{Overall approach}
Specifically, the normal map is defined using a unit vector $\normalmap{i,j} \in \mathbb{R}^3$ for each pixel $(i,j) \in \faceregion$. Noting that each face pixel corresponds to a normal vector facing towards the camera~\cite{Kemelmacher-ShlizermanB11}, we represent $\normalmap{i,j}$ using two variables $p_{i,j}, q_{i,j}$ as
\begin{equation}
	\normalmap{i,j} = \frac{(p_{i,j},q_{i,j},-1)}{\sqrt{p_{i,j}^2+q_{i,j}^2+1}}.
	\label{eq:pixelnormal}
\end{equation}
The values $\{p_{i,j}\}$, $\{q_{i,j}\}$ are computed by solving an optimization problem that will be explained later. The final height-field face model, represented using a depth value $z_{i,j}$ per pixel, is then determined so that the height field normals are as close as possible to the normal map. We note that the height field normal $\heightfieldnormal{i,j}$ at pixel $(i,j)$ can be computed using three points $\mathbf{h}_{i,j} = (i, j, z_{i,j})$, $\mathbf{h}_{i,j+1} = (i, j+1, z_{i, j+1})$, $\mathbf{h}_{i+1,j} = (i+1, j, z_{i+1,j})$ on the height field surface via
\begin{eqnarray}
	\heightfieldnormal{i,j} &=& \frac{ (\mathbf{h}_{i,j+1}  - \mathbf{h}_{i,j}) \times (\mathbf{h}_{i+1,j}  - \mathbf{h}_{i,j}) }
			{\|(\mathbf{h}_{i,j+1}  - \mathbf{h}_{i,j}) \times (\mathbf{h}_{i+1,j}  - \mathbf{h}_{i,j})\|_2} \nonumber\\
			&=& \frac{\left(z_{i+1,j}-z_{i,j}, z_{i,j+1}-z_{i,j}, -1\right)}{\sqrt{(z_{i+1,j}-z_{i,j})^2+(z_{i,j+1}-z_{i,j})^2+1}}.
\end{eqnarray}
Comparing this with Eq.~\eqref{eq:pixelnormal} shows that for the height field normal to be consistent with the normal map, we should have
\begin{equation}
	z_{i+1,j}-z_{i,j} = p_{i,j}, \quad z_{i,j+1}-z_{i,j} = q_{i,j}
	\label{eq:HeightFieldConditions}
\end{equation}
for every pixel. As these conditions only determine $\{z_{i,j}\}$ up to an additional constant, we compute $\{z_{i,j}\}$ as the minimum-norm solution to a linear least-squares problem
\begin{equation}
	\min_{\{z_{i,j}\}} ~~ \sum_{(i,j)} (z_{i+1,j}-z_{i,j} - p_{i,j})^2 + (z_{i,j+1}-z_{i,j} - q_{i,j})^2.
	\label{eq:DepthReconstruction}
\end{equation}

\paraheading{Normal map optimization}
For high-quality results, we enforce certain desirable properties of the computed normal map $\normalmap{i,j}$ by minimizing an energy that corresponds to these properties. First of all, the normal map should capture fine-scale details from the input image. Using the lighting and albedo parameters obtained during the computation of the medium face, we can evaluate the pixel intensity values from the normal map according to Eq.~\eqref{eq:sphericalharmoniclighting}, and require them to be close to the input image. However, such direct approach can suffer from the inaccuracy of spherical harmonics in complex lighting conditions such as cast shadows, which can lead to unsatisfactory results. Instead, we aim at minimizing the difference in intensity gradients, between the input image and the shading from the normal map. This difference can be measured using the following energy
\begin{equation}
	\egrad = \sum_{(i,j)} \left\| 
	\left[
	\begin{matrix}
	s'_{i+1,j}-s'_{i,j}\\
	s'_{i,j+1}-s'_{i,j}\\
	\end{matrix}
	\right] \!\!-\!\!
	\left[
	\begin{matrix}
	I_{i+1,j}-I_{i,j}\\
	I_{i,j+1}-I_{i,j}\\
	\end{matrix} 
	\right]
	\right\|_2^2,
	\label{eq:GradientEnergy}
\end{equation}
where $\{I_{i,j}\}$ are intensity values from the input image, and
\begin{equation}
	s'_{i,j} = r_{i,j} \cdot \max(\sphcoef^T \bfH(\normalmap{i,j}), 0)
\end{equation}
are shading intensities for the normal map according to Eq.~\eqref{eq:sphericalharmoniclighting}, using the optimized albedo $\{r_{i,j}\}$ and spherical harmonic coefficients $\sphcoef$ from Sec.~\ref{sec:medium}. Minimizing the difference in gradients instead of intensities helps to attenuate the influence from illumination noises such as cast shadows, while preserving the features from the input image. \revise{Another benefit is that its optimality condition is a higher-order PDE that results in smoother solution and reduces unnatural sharp features~\cite{Lysaker2003}. One example is shown in Fig.~\ref{fig:shadow_cast}, where the formulation with gradient difference reduces the sharp creases around the nose and the mouth.
 (see Fig.~\ref{fig:shadow_cast})}.

Optimizing $\egrad{}$ alone is not sufficient for good results, since the problem is under-constrained. Thus we introduce two additional regularization terms for the normal map. First we note that the medium face model from Sec.~\ref{sec:medium} provides good approximation of the final shape. Thus we introduce the following energy to penalize the deviation between normal map and the normals from the medium face  
\begin{equation}
	\eclose  = \sum_{(i,j)} \| \normalmap{i,j} - \bfn_{i,j} \|_2^2,
\end{equation}
where $\bfn_{i,j}$ is computed from the medium face mesh according to Eq.~\eqref{eq:meshnormal}. In addition, we enforce smoothness of the normal map using an energy that penalizes its gradient
\begin{equation}
	\esmooth = \sum_{(i,j)} \| \normalmap{i+1,j} - \normalmap{i,j} \|_2^2 + \| \normalmap{i,j+1} - \normalmap{i,j} \|_2^2.
\end{equation}

Finally, we need to ensure the normal map is \emph{integrable}, i.e., given the normal map there exists a height field surface such that conditions~\eqref{eq:HeightFieldConditions} are satisfied. Note that if \eqref{eq:HeightFieldConditions} are satisfied, then $p_{i,j}$ and $q_{i,j}$ are the increments of function $z$ along the grid directions. Moreover, the total increment of $z$ along the close path that connects pixels $(i,j), (i+1, j), (i+1, j+1), (i, j+1)$ should be zero, which results in the condition
\begin{equation}
	p_{i,j} + q_{i+1, j} - p_{i, j+1} - q_{i,j} = 0.
	\label{eq:IngegrabilityCondition}
\end{equation}
For the normal map to be integrable, this condition should be satisfied at each pixel. Indeed, with condition~\eqref{eq:HeightFieldConditions} we can interpret $p$ and $q$ as partial derivatives $\frac{\partial z}{\partial u}, \frac{\partial z}{\partial v}$ where $u, v$ are the grid directions; then condition~\eqref{eq:IngegrabilityCondition} corresponds to $\frac{\partial p}{\partial v} = \frac{\partial q}{\partial u}$, which is the condition for $(p, q)$ to be a gradient field. We can then enforce the integrability condition using an energy
\begin{equation}
	\eint = \sum_{(i,j)} ( p_{i,j} + q_{i+1, j} - p_{i, j+1} - q_{i,j} )^2.
	\label{eq:IntegrabilityEnergy}
\end{equation}
Combining the above energies, we derive an optimization problem for computing the desirable normal map
\begin{equation}
	\min_{\mathbf{p}, \mathbf{q}} \quad \egrad + \weightnormalmap{1} \eclose + \weightnormalmap{2} \esmooth + \weightnormalmap{3} \eint,
	\label{eq:normalmapoptimization}
\end{equation}  
where the optimization variables $\mathbf{p}, \mathbf{q}$ are the values $\{p_{i,j}\}, \{ q_{i,j} \}$, and $\weightnormalmap{1}, \weightnormalmap{2}, \weightnormalmap{3}$ are user-specified weights. This nonlinear least-squares problem is again solved using the Levenberg-Marquardt algorithm.

Fig.~\ref{fig:enhance} shows a fine face model reconstructed using our method. Compared with the medium face model, it captures more geometric details and reduces the reconstruction error. Besides, it can be observed from the reconstruction results in last two columns that the initial coarse face model has a large influence on reconstruction accuracy.

\begin{figure}[t] 
   \centering
      \includegraphics[width=\columnwidth]{show_cast.pdf}
   \caption{\revise{An input image with cast shadow and noise (a), and its reconstruction results by minimizing the intensity difference (b) and the gradient difference (c), respectively. Compared with intensity difference minimization, the formulation with gradient difference produces a smoother result and reduces unnatural sharp creases at the eye, the nose, and the mouth (highlighted with rectangles).}}
  \label{fig:shadow_cast}
 \end{figure}

%\bailin{The following paragraphs are only kept for comparison}
%Based on the shading cues and the medium face model, it is confident to extract a good photometric normal map from the input image $\bfI(u, v)$. However, the normals estimation becomes more difficult because of the nonlinear constraint that $n_x^2 + n_y^2 + n_z^2 = 1$. As observed in Ira~\cite{Kemelmacher-ShlizermanB11}, only normals facing the camera are observed. In our experiments, we also find that almost all the normals of the front part are towards to the negative z-axis in camera coordinate system. So we can write a normal $\bfn(u, v)$ as
%\begin{equation}\label{eq:normal}
%\bfn(u, v) = \frac{(p,q,-1)}{\sqrt{p^2+q^2+1}},
%\end{equation}
%where $p(u, v) = -\frac{n_x}{n_z}$, $q(u, v) = -\frac{n_y}{n_z}$, and the unit length constrain can be removed. Following~\cite{roth2016adaptive}, we can optimize the enhancement energy over normals $\bfn$ with the fixed albedo $\bfr$ and harmonics coefficients $\bm{\xi}$. This is a direct way that may suffer from the illumination noise. However, there is a more robust way that is to minimize the shading gradient errors of the observed and computed irradiance values. By using the gradients, we can eliminate the influence of noise on the constant term of spherical harmonics at least. The gradient energy can be formulated as
%\begin{equation}\label{eq:the gradient energy}
%E_{gradient} \! = \!\! \sum_{i,j}
%\!\! \left\|\left[
%\begin{matrix}
%r_{i+1,j}s_{i+1,j}-r_{i,j}s_{i,j}\\
%r_{i,j+1}s_{i,j+1}-r_{i,j}s_{i,j}\\
%\end{matrix}
%\right] \!\!-\!\!
%\left[
%\begin{matrix}
%I_{i+1,j}-I_{i,j}\\
%I_{i,j+1}-I_{i,j}\\
%\end{matrix}
%\right]\right\|_2^2.
%\end{equation}
%with Eq.~\eqref{eq:normal}, the spherical harmonics $\bfH(\bfn)$ can be written as
%\begin{equation}\label{eq:the another formulation of second spherical harmonics}
%\bfH(\bfn) \! = \! [1, \frac{p}{N}, \frac{q}{N}, \frac{-1}{N}, \frac{pq}{N^2}, \frac{-p}{N^2}, \frac{-q}{N^2}, \frac{p^2-q^2}{N^2}, \frac{3}{N^2} \! - \! 1]^T,
%\end{equation}
%where $N = \sqrt{p^2+q^2+1}$. Thus the illumination 
%\begin{equation}\label{eq:the another formulation of illumination}
%\begin{aligned}
%s_{i,j}= \xi_0 \! &+ \! \frac{1}{N}(\xi_1p_{i,j} \! + \! \xi_2q_{i,j} \! - \! \xi_3) \\
%&+ \! \frac{1}{N^2}(\xi_4p_{i,j}q_{i,j} \! - \! \xi_5p_{i,j} \! - \! \xi_6q_{i,j} \! + \! \xi_7(p_{i,j}^2 \! - \! q_{i,j}^2) \! + \! 3\xi_8) \\
%&-\xi_8,
%\end{aligned}
%\end{equation}
%where the harmonics coefficients $\bm{\xi} = (\xi_0, \xi_1, ... , \xi_8)^T$. As in Eq.~\eqref{eq:the enhancemment energy}, we require that the illumination $s_{i,j}$ is non-negative.
%
%At this stage, the photometric normals estimation is still an under-constrained problem. To further improve the gradient energy, some additional conditions are required. Since we have a medium face model that is a good approximation of a real face shape in the input image, we want the estimated normals close to the initial normals. This closing energy can be formulated as
%\begin{equation}\label{eq:the closing energy}
%E_{closing} = \lambda_{f1} \left\|\bfN - \bfN_0 \right\|_F^2,
%\end{equation}
%where $\bfN$ and $\bfN_0$ stack all the estimated normals and initial normals respectively. The closing energy can guarantee the smoothness of estimated normals and avoidance of very sharp features for clean images. However, we also need formulate a regularization term to reduce noise 
%\begin{equation}\label{eq:the reularization}
%E_{reg} = \lambda_{f2} \left\| \bfG\bfN  \right\|_F^2 + \lambda_{f3} \left\| \nabla \times \bfN \right\|_F^2,
%\end{equation}
%where $\nabla=(\frac{\partial}{\partial x}, \frac{\partial}{\partial y}, \frac{\partial}{\partial z})$, and $\times$ is the cross product. The last term in the regularization means that the estimated normals are integrable by penalizing the curl of the normal vector field.
%
%After the medium face modeling, we estimate a new normal for each pixel $\{i,j\}$ in the face region of the input image by optimizing
%\begin{equation}\label{eq:extracting normals}
%\min_{\bfp, \bfq} E_{gradient} + E_{closing} + E_{reg},
%\end{equation}
%where $\bfp$ stack all the $p_{i,j}$, $\bfq$ stack all the $q_{i,j}$. We use a Gauss-Newton approach to solve this nonlinear least square problem.
%
%\noindent\textbf{Details Recovery}
%At last, we recover a depth value $z_{i,j}$ for each pixel $\{i, j\}$ in the face region of the input image. Given a dense normal field, there are many surface-from-dense-normals algorithms~\cite{frankot1988method, kovesi2005shapelets} to reconstruct a surface and we choose a simple and effective method which is similar to~\cite{wu2006dense}. This method requires that the normals of the reconstruction should be close to the estimated normals when all integration paths are considered. The normal of the face reconstruction at pixel $\{i, j\}$ can be estimated as
%\begin{equation}\label{eq:normal estimation of reconstruction}
%\begin{aligned}
%\hat{n}_{i,j}&=\frac{\left(1, 0, z_{i,j+1}-z_{i,j}\right) \times \left(0, -1, z_{i-1,j}-z_{i,j}\right)}{\left\|\left(1, 0, z_{i,j+1}-z_{i,j}\right) \times \left(0, -1, z_{i-1,j}-z_{i,j}\right)\right\|_2}\\
%&=\frac{\left(z_{i,j+1}-z_{i,j}, z_{i,j}-z_{i-1,j}, -1\right)}{\sqrt{(z_{i,j+1}-z_{i,j})^2+(z_{i,j}-z_{i-1,j})^2+1}},
%\end{aligned}
%\end{equation}
%Compared with Eq.~\eqref{eq:normal}, We can reconstruct the fine face model by optimizing
%\begin{equation}\label{eq:surface reconstruction}
%\min_{\bfz} \sum_{i,j} \left( \left\| z_{i,j+1} - z_{i,j} - p_{i,j} \right\|_2^2 + \left\| z_{i-1,j} - z_{i,j} + q_{i,j} \right\|_2^2 \right),
%\end{equation}
%This problem is a linear least square problem and can be solved easily. Finally we can obtain a face shape with fine-scale details. Fig.~\ref{fig:enhance:err3} shows the reconstruction error map of the fine face shape. After the fine-level optimization step, the mean error is further reduced to $4.67\%$.

\section{Experiments}
\label{sec:experiments}
This section presents experimental results, and compares our method with some existing approaches.

\begin{figure*}[!t]
	\centering
	\includegraphics[width=\textwidth]{exp1_1.pdf}
	\caption{Facial reconstruction from images of frontal pose and neutral expression. For each input image, we show the ground truth (GT) as well as the results using out method and the method from~\cite{ZhuLYYL15}, each in two viewpoints. We also show the error maps (according to Eq.~\eqref{eq:ErrorMap}) for the two methods, together with their 3DRMSE.}
	\label{fig:exprement_A_8subjects}
\end{figure*}

\paraheading{Experimental setup}
To verify the effectiveness of our method, we tested it using the data set from the Bosphorus database~\cite{savran2008bosphorus}. This database provides structured-light scanned 3D face point clouds for 105 subjects, as well as their corresponding single-view 2D face photographs. For each subject, the database provides point clouds and images for different facial expressions and head poses. We ran our algorithm on the 2D images, and used the corresponding point clouds as ground truth to evaluate the reconstruction error. 55 subjects with low noises in their point clouds were chosen for testing. The reconstructed face is aligned with its corresponding ground truth face using iterative closest point (ICP) method~\cite{RusinkiewiczL01}. After alignment, we crop the face model at a radius of 85mm around the tip of the nose, and then compute the 3D Root Mean Square Error (3DRMSE):
\begin{equation}
	\sqrt{\sum_{i}(\bfX-\bfX^*)^2}/N,
	\label{eq:ErrorMap}
\end{equation}
where $\bfX$ is the reconstructed face, $\bfX^*$ is the grund truth, $N$ is the \revise{number of vertices} of the cropped frontal reconstructed face. We also computed the mean and standard deviation of all these errors.

\begin{figure*}[!t]
	\centering
	\includegraphics[width=0.9\textwidth]{exp2_3.pdf}
	\caption{Face reconstructions of four subjects from images of frontal pose with different expressions (happy, surprise, disgust), and of different poses (Yaw $+10^{\circ}$, $+20^{\circ}$, $+30^{\circ}$) with neutral expression. For each input image, we show the reconstructed face mesh as well as its textured rendering.}
	\label{fig:exprement_B_4_subjects}
\end{figure*}

Our algorithm is implemented in C++ and is tested on a PC with an Intel Core i7-4710MQ 2.50 GHz CPU and 7.5 GB RAM. The weights in optimization problems \eqref{eq:2D3DFitting}, \eqref{eq:AlbedoLightingOptimization}, \eqref{eq:ReducedDeformationOptimization}, \eqref{eq:normalmapoptimization} are set as follows: $\weightfit{1} = \weightfit{2} = 1.5 \times 10^3$;$\weightalbedo{1} = 5$;$\weightsubspace = 20$; $\weightnormalmap{1} = 10, \weightnormalmap{2} = 10, \weightnormalmap{3} = 1$. The nonlinear least-squares problems are solved using the \ceres{} solver~\cite{ceres-solver}, with all derivatives evaluated using automatic differentiation. To speed up the algorithm, we downsample the high-resolution 2D images from the database to $30\%$ of their original dimensions before running our algorithm. The down-sampled images have about $400 \times 500$ pixels, for which the coarse, medium, and fine face construction steps take about 1 second, 2 minutes, and 1 minute respectively using our non-optimized implementation.
\begin{table}[tbp]
\centering
\caption{The mean and standard variation of our reconstructions for each pose and expression.}
\vspace{1.5mm}
\begin{tabular}{lccc}
\hline
Pose &Yaw $+10^{\circ}$ &Yaw $+20^{\circ}$ &Yaw $+30^{\circ}$\\ \hline      
3DRMSE &$1.73\pm0.33$ &$1.51\pm0.24$ &$1.44\pm0.32$\\ \hline
Expression &happy &surprise &disgust\\ \hline             
3DRMSE &$1.71\pm0.34$ &$2.05\pm0.49$ &$1.98\pm0.42$\\ \hline
\end{tabular}
\label{pose_exp}
\end{table}

\paraheading{Frontal and neutral faces}
We first tested our method on facial images of frontal pose and neutral expression, from 55 subjects in the Bosphorus database. For comparison we also ran the face reconstruction method from~\cite{ZhuLYYL15}, which is based on a 3DMM built from \bfm{} and \facewarehouse{}.
Fig.~\ref{fig:exprement_A_8subjects} presents the reconstruction results of six subjects using our method and~\cite{ZhuLYYL15}, and compares them with the ground truth faces. Thanks to the enhancement in the medium face step and the SFS recovery in the fine face step, our approach can not only obtain a more realistic global facial shape, but also accurately capture the person-specific geometric details such as wrinkles. Fig.~\ref{fig:exprement_A_8subjects} also shows the 3DRMSE for our results and the results using~\cite{ZhuLYYL15}. The mean and standard variation of 3DRMSE is $1.97\pm0.35$ for the results by method~\cite{ZhuLYYL15}, and $1.56\pm0.24$ for the results by our method. It can be seen that the mean error from our results are consistently lower than those from the method of~\cite{ZhuLYYL15}.
 \begin{figure*}[!t] 
	\centering
	\includegraphics[width=2\columnwidth]{micc.pdf}
	\caption{Face reconstructions of four subjects from the MICC dataset~\cite{Bagdanov2011} using our method. We show from left to right the input image, the ground truth, our \revise{reconstruction result (with texture)} in two view points, and error map (according to Eq.~\eqref{eq:ErrorMap}).}
	\label{fig:micc_results}
\end{figure*}

\revise{\paraheading{Near-frontal poses and expressions}}
We also tested our method on face images with \revise{near-frontal} poses and expressions. First, for each of the 55 subjects, we applied our method on their images of neutral expression with three types of poses: Yaw $+10^{\circ}$, $+20^{\circ}$, and $+30^{\circ}$. Then, we tested our approach on frontal faces with three non-neutral expressions: happy, surprise, and disgust. Among the 55 subjects, there are 25 of them with all three expressions present. We apply our method on these 25 subjects, and Table~\ref{pose_exp} shows the mean and standard deviation of 3DRMSE for each pose and expression. We can observe that the reconstruction results by our method are consistent for different poses and expressions, and the reconstruction errors are small. This is verified in Fig.~\ref{fig:exprement_B_4_subjects}, where we show the reconstruction results of four subjects under different poses and expressions.

Furthermore, using landmark detection methods designed for facial images with large pose (e.g., $90^\circ$), our approach can also reconstruct the 3D model well for such images. \revise{Two examples} are shown in Fig.~\ref{fig:bigpose}, where the landmarks are detected using the method from~\cite{zhu2016face}.

\begin{figure}[t] 
   \centering
      \includegraphics[width=0.8\columnwidth]{bigpose.pdf}
   \caption{\revise{Face reconstructions of face images with very large pose using our method. We show from left to right the input image, and the reconstruction result from two viewpoints.}}
  \label{fig:bigpose}
 \end{figure}
\paraheading{Unconstrained facial images}
To demonstrate the robustness of our approach on general unconstrained facial images, we compare our method with the \emph{structure from motion} (SFM) method~\cite{hernandez2017accurate} and the learning-based method~\cite{tran2016regressing} using the MICC dataset~\cite{Bagdanov2011}. The MICC dataset contains 53 video sequences of varying resolution, conditions and zoom levels for each subject, which is recorded in controlled, less controlled or uncontrolled environment. There is a structured-light scanning for each subject as the ground truth, and the reconstruction errors of the reconstruction results are computed following the way described in the above.
\begin{table}[tbp]
\centering
\caption{Quantitative results on the MICC dataset~\cite{Bagdanov2011}. The mean and standard variation of 3DRMSE, the runtimes.}
\vspace{1.5mm} 
\begin{tabular}{lccc}
\hline
Approach &3DRMSE &run time\\ \hline      
SFM~\cite{hernandez2017accurate} &$1.92\pm0.39$ &CPU 1min 13s\\
CNN-based methods~\cite{tran2016regressing} &$1.53\pm0.29$ &GPU 0.088s\\ 
Ours &$1.75\pm0.29$ &CPU 3min\\ \hline
\end{tabular}
\label{micc}
\end{table}
For each subject, we select the most frontal face image from the corresponding outdoor video and reconstruct the 3D face model by setting it as input. Table~\ref{micc} shows that our reconstruction error is close to~\cite{tran2016regressing} and lower than~\cite{hernandez2017accurate}. With the prior of reliable medium face and SFS recovery, our approach can also have good estimations on unconstrained \revise{images}. Fig.~\ref{fig:micc_results} presents the reconstruction results of four subjects using our method.

\begin{figure*}[!t] 
	\centering
	\includegraphics[height=0.95\textheight]{unconstrained.pdf}
	\caption{Face reconstructions from unconstrained images, using the method from~\cite{Kemelmacher-ShlizermanB11} and our method.}
	\label{fig:unconstrained}
\end{figure*}
We also compared our method with the SFS approach of~\cite{Kemelmacher-ShlizermanB11} on more general unconstrained facial images. 
Since there are no ground truth shapes for these images, we only compared them visually. For reliable comparison, we directly ran our algorithm on the example images provided in~\cite{Kemelmacher-ShlizermanB11}. Fig.~\ref{fig:unconstrained} presents the comparison results, showing both the reconstructed face geometry and its textured display. We can see that our approach produced more accurate reconstruction of the overall shape, and recovered more geometrical details such as winkles and teeth. Although both methods perform SFS reconstruction, there is major difference on how the shape and illumination priors are derived. In~\cite{Kemelmacher-ShlizermanB11} a reference face model is utilized as the shape prior to estimate illumination and initialize photometric normals; as the reference face model is not adapted to the target face shape, this can lead to unsatisfactory results. In comparison, with our method the medium face model is optimized to provide reliable estimates of the target shape and illumination, which enables more accurate reconstruction. 

\vspace{-2mm}
\section{Discussion and Conclusion}
\label{sec:Conclusion}

\revise{The main limitation of our method is that its performance for a given image depends on how well the overall face shape is covered by our constructed face model. This is because medium and fine face modeling have little effect on the coarse face shape; thus in order to achieve good results, the coarse face model needs to be close enough to the ground-truth overall shape, which can be achieved if the ground-truth face is close to the space spanned by our linear face model. By combining \facewarehouse{} and \bfm{} to construct the face model, our approach achieves good results on a large number of images. But for faces with large deviation from both \facewarehouse{} and \bfm{}, our method may not work well. One potential future work is to improve the face model by incorporating a larger variety of face datasets.}

\revise{Since we compute pixel values by multiplying albedo with lighting, there is an inherent ambiguity in determining albedo and lighting from given pixel values. Our approach alleviates the problem by using PCA albedo and second-order spherical harmonics lighting, but it does not fully resolve the ambiguity. Nevertheless, as we only intend to recover face geometry, such approach is sufficient for achieving good results.}

In this paper, we present a coarse-to-fine method to reconstruct a high-quality 3D face model from a single image. Our approach uses a bilinear face model and local corrective deformation fields to obtain a reliable initial face shape with large- and medium-scale features, which enables robust shape-from-shading reconstruction of fine facial details. The experiments demonstrate that our method can accurately reconstruct 3D face models from images with different poses and expressions, and recover the fine-scale geometrical details such as wrinkles and teeth. Our approach combines the benefits of low-dimensional face models and shape-from-shading, enabling more accurate and robust reconstruction. 

\vspace{-3mm}
% use section* for acknowledgment
\section*{Acknowledgments}
We would like to thank the reviewers for their time spent on reviewing our manuscript and their insightful comments helping us improving the article. This work was supported by the National Key R\&D Program of China (No. 2016YFC0800501), the National Natural Science Foundation of China (No. 61672481, No. 61672482 and No. 11626253), the Youth Innovation Promotion Association of CAS, and the One Hundred Talent Project of the Chinese Academy of Sciences.


%The authors would like to thank...


% Can use something like this to put references on a page
% by themselves when using endfloat and the captionsoff option.
%\ifCLASSOPTIONcaptionsoff
%  \newpage
%\fi



% trigger a \newpage just before the given reference
% number - used to balance the columns on the last page
% adjust value as needed - may need to be readjusted if
% the document is modified later
%\IEEEtriggeratref{8}
% The "triggered" command can be changed if desired:
%\IEEEtriggercmd{\enlargethispage{-5in}}

% references section

% can use a bibliography generated by BibTeX as a .bbl file
% BibTeX documentation can be easily obtained at:
% http://www.ctan.org/tex-archive/biblio/bibtex/contrib/doc/
% The IEEEtran BibTeX style support page is at:
% http://www.michaelshell.org/tex/ieeetran/bibtex/
%\bibliographystyle{IEEEtran}
% argument is your BibTeX string definitions and bibliography database(s)
%\bibliography{IEEEabrv,../bib/paper}
%
% <OR> manually copy in the resultant .bbl file
% set second argument of \begin to the number of references
% (used to reserve space for the reference number labels box)
%\begin{thebibliography}{1}

%\bibitem{IEEEhowto:kopka}
%H.~Kopka and P.~W. Daly, \emph{A Guide to \LaTeX}, 3rd~ed.\hskip 1em plus
% 0.5em minus 0.4em\relax Harlow, England: Addison-Wesley, 1999.

%\end{thebibliography}

% biography section
%
% If you have an EPS/PDF photo (graphicx package needed) extra braces are
% needed around the contents of the optional argument to biography to prevent
% the LaTeX parser from getting confused when it sees the complicated
% \includegraphics command within an optional argument. (You could create
% your own custom macro containing the \includegraphics command to make things
% simpler here.)
%\begin{IEEEbiography}[{\includegraphics[width=1in,height=1.25in,clip,keepaspectratio]{mshell}}]{Michael Shell}
% or if you just want to reserve a space for a photo:

%\begin{IEEEbiography}{Michael Shell}
%Biography text here.
%\end{IEEEbiography}

% if you will not have a photo at all:
%\begin{IEEEbiographynophoto}{John Doe}
%Biography text here.
%\end{IEEEbiographynophoto}

% insert where needed to balance the two columns on the last page with
% biographies
%\newpage

%\begin{IEEEbiographynophoto}{Jane Doe}
%Biography text here.
%\end{IEEEbiographynophoto}

% You can push biographies down or up by placing
% a \vfill before or after them. The appropriate
% use of \vfill depends on what kind of text is
% on the last page and whether or not the columns
% are being equalized.

%\vfill

% Can be used to pull up biographies so that the bottom of the last one
% is flush with the other column.
%\enlargethispage{-5in}
\bibliographystyle{IEEEtran}
%%use following if all content of bibtex file should be shown
%\nocite{*}
\bibliography{SingleImageReconstruction}

\begin{IEEEbiography}[{\includegraphics[width=1in]{JiangLuo.jpg}}]{Luo Jiang}
is currently working towards the PhD degree at the University of Science and Technology of China.
He obtained his bachelor degree in 2013 from the Huazhong University of Science and Technol-
ogy, China. His research interests include computer graphics, image processing and deep learning.
\end{IEEEbiography}

\vspace{-5mm}
\begin{IEEEbiography}[{\includegraphics[width=1in]{Juyong.png}}]{Juyong Zhang}
is an associate professor in the School of Mathematical Sciences at University of Science and Technology of China. He received the BS degree from the University of Science and Technology of China in 2006, and the PhD degree from Nanyang Technological University, Singapore. His research interests include computer graphics, computer vision, and numerical optimization. He is an associate editor of The Visual Computer.
\end{IEEEbiography}

\vspace{-5mm}
\begin{IEEEbiography}[{\includegraphics[width=1in]{Bailin.png}}]{Bailin Deng}
is a lecturer in the School of Computer Science and Informatics at Cardiff University. He received the BEng degree in computer software (2005) and the MSc degree in computer science (2008) from Tsinghua University (China), and the PhD degree in technical mathematics from Vienna University of Technology (Austria). His research interests include geometry processing, numerical optimization, computational design, and digital fabrication. He is a member of the IEEE.
\end{IEEEbiography}


\vspace{-5mm} \begin{IEEEbiography}[{\includegraphics[width=1in]{LiHao.jpg}}]{Hao Li}
received the BSc degree in 2011 from the University of Science and Technology of China. His research interests include computer graphics and image processing.
\end{IEEEbiography}

\vspace{-10mm}
\begin{IEEEbiography}[{\includegraphics[width=1in]{Ligang.png}}]{Ligang Liu}
is a Professor at the School of Mathematical Sciences, University of Science and Technology of China. His research interests include digital geometric processing, computer graphics, and image processing. He serves as the associated editors for journals of IEEE Transactions on Visualization and Computer Graphics, IEEE Computer Graphics and Applications, Computer Graphics Forum, Computer Aided Geometric Design, and The Visual Computer. He served as the conference co-chair of GMP 2017 and the program co-chairs of GMP 2018, CAD/Graphics 2017, CVM 2016, SGP 2015, and SPM 2014. His research works could be found at his research website: http://staff.ustc.edu.cn/lgliu.
\end{IEEEbiography}

% that's all folks
\end{document}




SingleImageReconstruction_v2-full.bbl
% Generated by IEEEtran.bst, version: 1.14 (2015/08/26)
\begin{thebibliography}{10}
\providecommand{\url}[1]{#1}
\csname url@samestyle\endcsname
\providecommand{\newblock}{\relax}
\providecommand{\bibinfo}[2]{#2}
\providecommand{\BIBentrySTDinterwordspacing}{\spaceskip=0pt\relax}
\providecommand{\BIBentryALTinterwordstretchfactor}{4}
\providecommand{\BIBentryALTinterwordspacing}{\spaceskip=\fontdimen2\font plus
\BIBentryALTinterwordstretchfactor\fontdimen3\font minus
  \fontdimen4\font\relax}
\providecommand{\BIBforeignlanguage}[2]{{%
\expandafter\ifx\csname l@#1\endcsname\relax
\typeout{** WARNING: IEEEtran.bst: No hyphenation pattern has been}%
\typeout{** loaded for the language `#1'. Using the pattern for}%
\typeout{** the default language instead.}%
\else
\language=\csname l@#1\endcsname
\fi
#2}}
\providecommand{\BIBdecl}{\relax}
\BIBdecl

\bibitem{Stylianou2009}
G.~Stylianou and A.~Lanitis, ``Image based {3D} face reconstruction: A
  survey,'' \emph{International Journal of Image and Graphics}, vol.~9, no.~2,
  pp. 217--250, 2009.

\bibitem{BlanzV03}
V.~Blanz and T.~Vetter, ``Face recognition based on fitting a 3d morphable
  model,'' \emph{{IEEE} Trans. Pattern Anal. Mach. Intell.}, vol.~25, no.~9,
  pp. 1063--1074, 2003.

\bibitem{ZhuLYYL15}
X.~Zhu, Z.~Lei, J.~Yan, D.~Yi, and S.~Z. Li, ``High-fidelity pose and
  expression normalization for face recognition in the wild,'' in \emph{{IEEE}
  Conference on Computer Vision and Pattern Recognition}, 2015, pp. 787--796.

\bibitem{CaoHZ14}
C.~Cao, Q.~Hou, and K.~Zhou, ``Displaced dynamic expression regression for
  real-time facial tracking and animation,'' \emph{{ACM} Trans. Graph.},
  vol.~33, no.~4, pp. 43:1--43:10, 2014.

\bibitem{Thies_2016_CVPR}
J.~Thies, M.~Zollhöfer, M.~Stamminger, C.~Theobalt, and M.~Nießner,
  ``Face2face: Real-time face capture and reenactment of rgb videos,'' in
  \emph{{IEEE} Conference on Computer Vision and Pattern Recognition}, 2016,
  pp. 2387--2395.

\bibitem{BlanzV99}
V.~Blanz and T.~Vetter, ``A morphable model for the synthesis of 3d faces,'' in
  \emph{Proceedings of the 26th Annual Conference on Computer Graphics and
  Interactive Techniques, {SIGGRAPH}}, 1999, pp. 187--194.

\bibitem{Zhang99shapefrom}
R.~Zhang, P.-S. Tsai, J.~E. Cryer, and M.~Shah, ``Shape-from-shading: a
  survey,'' \emph{IEEE Transactions on Pattern Analysis and Machine
  Intelligence}, vol.~21, no.~8, pp. 690--706, 1999.

\bibitem{Kemelmacher-ShlizermanB11}
I.~Kemelmacher-Shlizerman and R.~Basri, ``3d face reconstruction from a single
  image using a single reference face shape.'' \emph{IEEE Trans. Pattern Anal.
  Mach. Intell.}, vol.~33, no.~2, pp. 394--405, 2011.

\bibitem{CaoWZTZ14}
C.~Cao, Y.~Weng, S.~Zhou, Y.~Tong, and K.~Zhou, ``Facewarehouse: {A} 3d facial
  expression database for visual computing,'' \emph{{IEEE} Trans. Vis. Comput.
  Graph.}, vol.~20, no.~3, pp. 413--425, 2014.

\bibitem{Paysan2009}
P.~Paysan, R.~Knothe, B.~Amberg, S.~Romdhani, and T.~Vetter, ``A 3d face model
  for pose and illumination invariant face recognition,'' in \emph{2009 Sixth
  IEEE International Conference on Advanced Video and Signal Based
  Surveillance}, 2009, pp. 296--301.

\bibitem{Sirovich1987}
L.~Sirovich and M.~Kirby, ``Low-dimensional procedure for the characterization
  of human faces,'' \emph{Journal of the Optical Society of America A}, vol.~4,
  no.~3, pp. 519--524, 1987.

\bibitem{Meytlis2007}
M.~Meytlis and L.~Sirovich, ``On the dimensionality of face space,'' \emph{IEEE
  Transactions on Pattern Analysis and Machine Intelligence}, vol.~29, no.~7,
  pp. 1262--1267, 2007.

\bibitem{romdhani2005estimating}
S.~Romdhani and T.~Vetter, ``Estimating 3d shape and texture using pixel
  intensity, edges, specular highlights, texture constraints and a prior,'' in
  \emph{{IEEE} Conference on Computer Vision and Pattern Recognition}, vol.~2,
  2005, pp. 986--993.

\bibitem{keller20073d}
M.~Keller, R.~Knothe, and T.~Vetter, ``3d reconstruction of human faces from
  occluding contours,'' \emph{Computer Vision/Computer Graphics Collaboration
  Techniques}, pp. 261--273, 2007.

\bibitem{bas2016fitting}
A.~Bas, W.~A. Smith, T.~Bolkart, and S.~Wuhrer, ``Fitting a 3d morphable model
  to edges: A comparison between hard and soft correspondences,'' in
  \emph{Asian Conference on Computer Vision}.\hskip 1em plus 0.5em minus
  0.4em\relax Springer, 2016, pp. 377--391.

\bibitem{huber2016multiresolution}
P.~Huber, G.~Hu, R.~Tena, P.~Mortazavian, P.~Koppen, W.~J. Christmas,
  M.~Ratsch, and J.~Kittler, ``A multiresolution 3d morphable face model and
  fitting framework,'' in \emph{Proceedings of the 11th International Joint
  Conference on Computer Vision, Imaging and Computer Graphics Theory and
  Applications}, 2016.

\bibitem{Blanz2004-Exchange}
V.~Blanz, K.~Scherbaum, T.~Vetter, and H.-P. Seidel, ``Exchanging faces in
  images,'' \emph{Computer Graphics Forum}, vol.~23, no.~3, pp. 669--676, 2004.

\bibitem{Scherbaum2011}
K.~Scherbaum, T.~Ritschel, M.~Hullin, T.~ThormÃ¤hlen, V.~Blanz, and H.-P.
  Seidel, ``Computer-suggested facial makeup,'' \emph{Computer Graphics Forum},
  vol.~30, no.~2, pp. 485--492, 2011.

\bibitem{VlasicBPP05}
D.~Vlasic, M.~Brand, H.~Pfister, and J.~Popovic, ``Face transfer with
  multilinear models.'' \emph{ACM Trans. Graph.}, vol.~24, no.~3, pp. 426--433,
  2005.

\bibitem{Lewis2014-Blendshape}
J.~P. Lewis, K.~Anjyo, T.~Rhee, M.~Zhang, F.~Pighin, and Z.~Deng, ``Practice
  and theory of blendshape facial models,'' in \emph{Eurographics 2014 - State
  of the Art Reports}, S.~Lefebvre and M.~Spagnuolo, Eds.\hskip 1em plus 0.5em
  minus 0.4em\relax The Eurographics Association, 2014.

\bibitem{Weise2009}
T.~Weise, H.~Li, L.~Van~Gool, and M.~Pauly, ``Face/off: Live facial puppetry,''
  in \emph{Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on
  Computer Animation}.\hskip 1em plus 0.5em minus 0.4em\relax ACM, 2009, pp.
  7--16.

\bibitem{WeiseBLP11}
T.~Weise, S.~Bouaziz, H.~Li, and M.~Pauly, ``Realtime performance-based facial
  animation,'' \emph{{ACM} Trans. Graph.}, vol.~30, no.~4, p.~77, 2011.

\bibitem{BouazizWP13}
S.~Bouaziz, Y.~Wang, and M.~Pauly, ``Online modeling for realtime facial
  animation.'' \emph{ACM Trans. Graph.}, vol.~32, no.~4, pp. 40:1--40:10, 2013.

\bibitem{li2013realtime}
H.~Li, J.~Yu, Y.~Ye, and C.~Bregler, ``Realtime facial animation with
  on-the-fly correctives,'' \emph{ACM Transactions on Graphics (Proceedings
  SIGGRAPH 2013)}, vol.~32, no.~4, July 2013.

\bibitem{HsiehMYL15}
P.~Hsieh, C.~Ma, J.~Yu, and H.~Li, ``Unconstrained realtime facial performance
  capture,'' in \emph{{IEEE} Conference on Computer Vision and Pattern
  Recognition}, 2015, pp. 1675--1683.

\bibitem{CaoWLZ13}
C.~Cao, Y.~Weng, S.~Lin, and K.~Zhou, ``3d shape regression for real-time
  facial animation.'' \emph{ACM Trans. Graph.}, vol.~32, no.~4, 2013.

\bibitem{CaoBZB15}
C.~Cao, D.~Bradley, K.~Zhou, and T.~Beeler, ``Real-time high-fidelity facial
  performance capture,'' \emph{{ACM} Trans. Graph.}, vol.~34, no.~4, 2015.

\bibitem{Gross2005}
R.~Gross, ``Face databases,'' in \emph{Handbook of Face Recognition}.\hskip 1em
  plus 0.5em minus 0.4em\relax Springer New York, 2005, pp. 301--327.

\bibitem{Durou2008}
J.-D. Durou, M.~Falcone, and M.~Sagona, ``Numerical methods for
  shape-from-shading: A new survey with benchmarks,'' \emph{Computer Vision and
  Image Understanding}, vol. 109, no.~1, pp. 22 -- 43, 2008.

\bibitem{Prados2006}
E.~Prados and O.~Faugeras, ``Shape from shading,'' in \emph{Handbook of
  Mathematical Models in Computer Vision}, N.~Paragios, Y.~Chen, and
  O.~Faugeras, Eds.\hskip 1em plus 0.5em minus 0.4em\relax Springer US, 2006,
  pp. 375--388.

\bibitem{Shimshoni2000}
I.~Shimshoni, Y.~Moses, and M.~Lindenbaum, ``Shape reconstruction of 3d
  bilaterally symmetric surfaces,'' \emph{International Journal of Computer
  Vision}, vol.~39, no.~2, pp. 97--110, 2000.

\bibitem{Zhao2000}
W.~Y. Zhao and R.~Chellappa, ``Illumination-insensitive face recognition using
  symmetric shape-from-shading,'' in \emph{{IEEE} Conference on Computer Vision
  and Pattern Recognition}, vol.~1, 2000, pp. 286--293.

\bibitem{Zhao2001}
Zhao, Wen Yi and Chellappa, Rama, ``Symmetric shape-from-shading using self-ratio image,''
  \emph{International Journal of Computer Vision}, vol.~45, no.~1, pp. 55--75,
  2001.

\bibitem{Atick1996}
J.~J. Atick, P.~A. Griffin, and A.~N. Redlich, ``Statistical approach to shape
  from shading: Reconstruction of three-dimensional face surfaces from single
  two-dimensional images,'' \emph{Neural Computation}, vol.~8, pp. 1321--1340,
  1996.

\bibitem{Dovgard2004}
R.~Dovgard and R.~Basri, ``Statistical symmetric shape from shading for 3d
  structure recovery of faces,'' in \emph{ECCV}, T.~Pajdla and J.~Matas, Eds.,
  2004, pp. 99--113.

\bibitem{WuWMT11}
C.~Wu, B.~Wilburn, Y.~Matsushita, and C.~Theobalt, ``High-quality shape from
  multi-view stereo and shading under general illumination,'' in \emph{{IEEE}
  Conference on Computer Vision and Pattern Recognition}, 2011, pp. 969--976.

\bibitem{IchimBP15}
A.~E. Ichim, S.~Bouaziz, and M.~Pauly, ``Dynamic 3d avatar creation from
  hand-held video input.'' \emph{ACM Trans. Graph.}, vol.~34, no.~4, 2015.

\bibitem{Kemelmacher-Shlizerman2011-InTheWild}
I.~Kemelmacher-Shlizerman and S.~M. Seitz, ``Face reconstruction in the wild,''
  in \emph{International Conference on Computer Vision}, 2011.

\bibitem{RothTL15}
J.~Roth, Y.~Tong, and X.~Liu, ``Unconstrained 3d face reconstruction,'' in
  \emph{{IEEE} Conference on Computer Vision and Pattern Recognition}, 2015.

\bibitem{roth2016adaptive}
J.~Roth, Y.~Tong, and X.~Liu, ``Adaptive 3d face reconstruction from unconstrained photo
  collections,'' in \emph{{IEEE} Conference on Computer Vision and Pattern
  Recognition}, 2016, pp. 4197--4206.

\bibitem{sd2004}
R.~W. Sumner and J.~Popovi{\'c}, ``Deformation transfer for triangle meshes,''
  \emph{ACM Transactions on Graphics (TOG)}, vol.~23, no.~3, 2004.

\bibitem{de1997signal}
L.~De~Lathauwer, \emph{Signal processing based on multilinear algebra}.\hskip
  1em plus 0.5em minus 0.4em\relax Katholieke Universiteit Leuven, 1997.

\bibitem{ChenRWCS14}
D.~Chen, S.~Ren, Y.~Wei, X.~Cao, and J.~Sun, ``Joint cascade face detection and
  alignment,'' in \emph{{ECCV}}, 2014, pp. 109--122.

\bibitem{frolova2004accuracy}
D.~Frolova, D.~Simakov, and R.~Basri, ``Accuracy of spherical harmonic
  approximations for images of lambertian objects under far and near
  lighting,'' in \emph{ECCV}.\hskip 1em plus 0.5em minus 0.4em\relax Springer,
  2004, pp. 574--587.

\bibitem{strasser1974schnelle}
W.~Stra{\ss}er, ``Schnelle kurven-und fl{\"a}chendarstellung auf grafischen
  sichtger{\"a}ten,'' Ph.D. dissertation, 1974.

\bibitem{Zhang2010-Spectral}
H.~Zhang, O.~Van~Kaick, and R.~Dyer, ``Spectral mesh processing,''
  \emph{Computer Graphics Forum}, vol.~29, no.~6, pp. 1865--1894, 2010.

\bibitem{ValgaertsWBST12}
L.~Valgaerts, C.~Wu, A.~Bruhn, H.~Seidel, and C.~Theobalt, ``Lightweight
  binocular facial performance capture under uncontrolled lighting,''
  \emph{{ACM} Trans. Graph.}, vol.~31, no.~6, p. 187, 2012.

\bibitem{Madsen2004}
K.~Madsen, H.~B. Nielsen, and O.~Tingleff, ``Methods for non-linear least
  squares problems,'' Informatics and Mathematical Modelling, Technical
  University of Denmark, 2004, 2nd edition.

\bibitem{Lysaker2003}
M.~Lysaker, A.~Lundervold, and X.-C. Tai, ``Noise removal using fourth-order
  partial differential equation with applications to medical magnetic resonance
  images in space and time,'' \emph{IEEE Transactions on image processing},
  vol.~12, no.~12, pp. 1579--1590, 2003.

\bibitem{savran2008bosphorus}
A.~Savran, N.~Aly{\"u}z, H.~Dibeklio{\u{g}}lu, O.~{\c{C}}eliktutan,
  B.~G{\"o}kberk, B.~Sankur, and L.~Akarun, ``Bosphorus database for 3d face
  analysis,'' in \emph{European Workshop on Biometrics and Identity
  Management}, 2008.

\bibitem{RusinkiewiczL01}
S.~Rusinkiewicz and M.~Levoy, ``Efficient variants of the {ICP} algorithm,'' in
  \emph{3rd International Conference on 3D Digital Imaging and Modeling}, 2001,
  pp. 145--152.

\bibitem{ceres-solver}
S.~Agarwal, K.~Mierle, and Others, ``Ceres solver,''
  \url{http://ceres-solver.org}.

\bibitem{Bagdanov2011}
A.~D. Bagdanov, A.~Del~Bimbo, and I.~Masi, ``The florence 2d/3d hybrid face
  dataset,'' in \emph{Proceedings of the 2011 Joint ACM Workshop on Human
  Gesture and Behavior Understanding}, 2011, pp. 79--80.

\bibitem{zhu2016face}
X.~Zhu, Z.~Lei, X.~Liu, H.~Shi, and S.~Z. Li, ``Face alignment across large
  poses: A 3d solution,'' in \emph{{IEEE} Conference on Computer Vision and
  Pattern Recognition}, 2016, pp. 146--155.

\bibitem{hernandez2017accurate}
M.~Hernandez, T.~Hassner, J.~Choi, and G.~Medioni, ``Accurate 3d face
  reconstruction via prior constrained structure from motion,'' \emph{Computers
  \& Graphics}, 2017.

\bibitem{tran2016regressing}
A.~T. Tran, T.~Hassner, I.~Masi, and G.~G. Medioni, ``Regressing robust and
  discriminative 3d morphable models with a very deep neural network,'' in
  \emph{{IEEE} Conference on Computer Vision and Pattern Recognition}, 2017,
  pp. 1493--1502.

\end{thebibliography}
