  It is well known that popular optimization techniques can lead to overfitting
or even a lack of convergence altogether; thus, practitioners often utilize ad
hoc regularization terms added to the energy functional. When carefully
crafted, these regularizations can produce compelling results. However,
regularization changes both the energy landscape and the solution to the
optimization problem, which can result in underfitting. Surprisingly, many
practitioners both add regularization and claim that their model lacks the
expressivity to fit the data. Motivated by a geometric interpretation of the
linearized search space, we propose an approach that ameliorates overfitting
without the need for regularization terms that restrict the expressiveness of
the underlying model. We illustrate the efficacy of our approach on
minimization problems related to three-dimensional facial expression estimation
where overfitting clouds semantic understanding and regularization may lead to
underfitting that misses or misinterprets subtle expressions.
