  Recent advances in 3D vision have demonstrated the strengths of photometric
bundle adjustment. By directly minimizing reprojected pixel errors, instead of
geometric reprojection errors, such methods can achieve sub-pixel alignment
accuracy in both high and low textured regions. Typically, these problems are
solved using a forwards compositional Lucas-Kanade formulation parameterized by
6-DoF rigid camera poses and a depth per point in the structure. For large
problems the most CPU-intensive component of the pipeline is the creation and
factorization of the Hessian matrix at each iteration. For many warps, the
inverse compositional formulation can offer significant speed-ups since the
Hessian need only be inverted once. In this paper, we show that an ordinary
inverse compositional formulation does not work for warps of this type of
parameterization due to ill-conditioning of its partial derivatives. However,
we show that it is possible to overcome this limitation by introducing the
concept of a proxy template image. We show an order of magnitude improvement in
speed, with little effect on quality, going from forwards to inverse
compositional in our own photometric bundle adjustment method designed for
object-centric structure from motion. This means less processing time for large
systems or denser reconstructions under the same real-time constraints. We
additionally show that this theory can be readily applied to existing methods
by integrating it with the recently released Direct Sparse Odometry SLAM
algorithm.
