  We present an approach to backpropagating through minimal problem solvers in
end-to-end neural network training. Traditional methods relying on manually
constructed formulas, finite differences, and autograd are laborious,
approximate, and unstable for complex minimal problem solvers. We show that
using the Implicit function theorem (IFT) to calculate derivatives to
backpropagate through the solution of a minimal problem solver is simple, fast,
and stable. We compare our approach to (i) using the standard autograd on
minimal problem solvers and relate it to existing backpropagation formulas
through SVD-based and Eig-based solvers and (ii) implementing the backprop with
an existing PyTorch Deep Declarative Networks (DDN) framework. We demonstrate
our technique on a toy example of training outlier-rejection weights for 3D
point registration and on a real application of training an outlier-rejection
and RANSAC sampling network in image matching. Our method provides $100\%$
stability and is 10 times faster compared to autograd, which is unstable and
slow, and compared to DDN, which is stable but also slow.
