  We address the problem of learning observation models end-to-end for
estimation. Robots operating in partially observable environments must infer
latent states from multiple sensory inputs using observation models that
capture the joint distribution between latent states and observations. This
inference problem can be formulated as an objective over a graph that optimizes
for the most likely sequence of states using all previous measurements. Prior
work uses observation models that are either known a-priori or trained on
surrogate losses independent of the graph optimizer. In this paper, we propose
a method to directly optimize end-to-end tracking performance by learning
observation models with the graph optimizer in the loop. This direct approach
may appear, however, to require the inference algorithm to be fully
differentiable, which many state-of-the-art graph optimizers are not. Our key
insight is to instead formulate the problem as that of energy-based learning.
We propose a novel approach, LEO, for learning observation models end-to-end
with graph optimizers that may be non-differentiable. LEO alternates between
sampling trajectories from the graph posterior and updating the model to match
these samples to ground truth trajectories. We propose a way to generate such
samples efficiently using incremental Gauss-Newton solvers. We compare LEO
against baselines on datasets drawn from two distinct tasks: navigation and
real-world planar pushing. We show that LEO is able to learn complex
observation models with lower errors and fewer samples. Supplementary video:
https://youtu.be/YqzlUPudfkA
