  We're interested in the problem of estimating object states from touch during
manipulation under occlusions. In this work, we address the problem of
estimating object poses from touch during planar pushing. Vision-based tactile
sensors provide rich, local image measurements at the point of contact. A
single such measurement, however, contains limited information and multiple
measurements are needed to infer latent object state. We solve this inference
problem using a factor graph. In order to incorporate tactile measurements in
the graph, we need local observation models that can map high-dimensional
tactile images onto a low-dimensional state space. Prior work has used
low-dimensional force measurements or engineered functions to interpret tactile
measurements. These methods, however, can be brittle and difficult to scale
across objects and sensors. Our key insight is to directly learn tactile
observation models that predict the relative pose of the sensor given a pair of
tactile images. These relative poses can then be incorporated as factors within
a factor graph. We propose a two-stage approach: first we learn local tactile
observation models supervised with ground truth data, and then integrate these
models along with physics and geometric factors within a factor graph
optimizer. We demonstrate reliable object tracking using only tactile feedback
for 150 real-world planar pushing sequences with varying trajectories across
three object shapes. Supplementary video: https://youtu.be/y1kBfSmi8w0
