  RANSAC is an important algorithm in robust optimization and a central
building block for many computer vision applications. In recent years,
traditionally hand-crafted pipelines have been replaced by deep learning
pipelines, which can be trained in an end-to-end fashion. However, RANSAC has
so far not been used as part of such deep learning pipelines, because its
hypothesis selection procedure is non-differentiable. In this work, we present
two different ways to overcome this limitation. The most promising approach is
inspired by reinforcement learning, namely to replace the deterministic
hypothesis selection by a probabilistic selection for which we can derive the
expected loss w.r.t. to all learnable parameters. We call this approach DSAC,
the differentiable counterpart of RANSAC. We apply DSAC to the problem of
camera localization, where deep learning has so far failed to improve on
traditional approaches. We demonstrate that by directly minimizing the expected
loss of the output camera poses, robustly estimated by RANSAC, we achieve an
increase in accuracy. In the future, any deep learning pipeline can use DSAC as
a robust optimization component.
