  Fitting model parameters to a set of noisy data points is a common problem in
computer vision. In this work, we fit the 6D camera pose to a set of noisy
correspondences between the 2D input image and a known 3D environment. We
estimate these correspondences from the image using a neural network. Since the
correspondences often contain outliers, we utilize a robust estimator such as
Random Sample Consensus (RANSAC) or Differentiable RANSAC (DSAC) to fit the
pose parameters. When the problem domain, e.g. the space of all 2D-3D
correspondences, is large or ambiguous, a single network does not cover the
domain well. Mixture of Experts (MoE) is a popular strategy to divide a problem
domain among an ensemble of specialized networks, so called experts, where a
gating network decides which expert is responsible for a given input. In this
work, we introduce Expert Sample Consensus (ESAC), which integrates DSAC in a
MoE. Our main technical contribution is an efficient method to train ESAC
jointly and end-to-end. We demonstrate experimentally that ESAC handles two
real-world problems better than competing methods, i.e. scalability and
ambiguity. We apply ESAC to fitting simple geometric models to synthetic
images, and to camera re-localization for difficult, real datasets.
