  In this work, we derive a model for the covariance of the visual residuals in
multi-view SfM, odometry and SLAM setups. The core of our approach is the
formulation of the residual covariances as a combination of geometric and
photometric noise sources. And our key novel contribution is the derivation of
a term modelling how local 2D patches suffer from perspective deformation when
imaging 3D surfaces around a point. Together, these add up to an efficient and
general formulation which not only improves the accuracy of both feature-based
and direct methods, but can also be used to estimate more accurate measures of
the state entropy and hence better founded point visibility thresholds. We
validate our model with synthetic and real data and integrate it into
photometric and feature-based Bundle Adjustment, improving their accuracy with
a negligible overhead.
