  We propose and show the efficacy of a new method to address generic inverse
problems. Inverse modeling is the task whereby one seeks to determine the
control parameters of a natural system that produce a given set of observed
measurements. Recent work has shown impressive results using deep learning, but
we note that there is a trade-off between model performance and computational
time. For some applications, the computational time at inference for the best
performing inverse modeling method may be overly prohibitive to its use. We
present a new method that leverages multiple manifolds as a mixture of backward
(e.g., inverse) models in a forward-backward model architecture. These multiple
backwards models all share a common forward model, and their training is
mitigated by generating training examples from the forward model. The proposed
method thus has two innovations: 1) the multiple Manifold Mixture Network (MMN)
architecture, and 2) the training procedure involving augmenting backward model
training data using the forward model. We demonstrate the advantages of our
method by comparing to several baselines on four benchmark inverse problems,
and we furthermore provide analysis to motivate its design.
