  Many computer vision problems (e.g., camera calibration, image alignment,
structure from motion) are solved with nonlinear optimization methods. It is
generally accepted that second order descent methods are the most robust, fast,
and reliable approaches for nonlinear optimization of a general smooth
function. However, in the context of computer vision, second order descent
methods have two main drawbacks: (1) the function might not be analytically
differentiable and numerical approximations are impractical, and (2) the
Hessian may be large and not positive definite. To address these issues, this
paper proposes generic descent maps, which are average "descent directions" and
rescaling factors learned in a supervised fashion. Using generic descent maps,
we derive a practical algorithm - Supervised Descent Method (SDM) - for
minimizing Nonlinear Least Squares (NLS) problems. During training, SDM learns
a sequence of decent maps that minimize the NLS. In testing, SDM minimizes the
NLS objective using the learned descent maps without computing the Jacobian or
the Hessian. We prove the conditions under which the SDM is guaranteed to
converge. We illustrate the effectiveness and accuracy of SDM in three computer
vision problems: rigid image alignment, non-rigid image alignment, and 3D pose
estimation. In particular, we show how SDM achieves state-of-the-art
performance in the problem of facial feature detection. The code has been made
available at www.humansensing.cs.cmu.edu/intraface.
