  To perform complex tasks, robots must be able to interact with and manipulate
their surroundings. One of the key challenges in accomplishing this is robust
state estimation during physical interactions, where the state involves not
only the robot and the object being manipulated, but also the state of the
contact itself. In this work, within the context of planar pushing, we extend
previous inference-based approaches to state estimation in several ways. We
estimate the robot, object, and the contact state on multiple manipulation
platforms configured with a vision-based articulated model tracker, and either
a biomimetic tactile sensor or a force-torque sensor. We show how to fuse raw
measurements from the tracker and tactile sensors to jointly estimate the
trajectory of the kinematic states and the forces in the system via
probabilistic inference on factor graphs, in both batch and incremental
settings. We perform several benchmarks with our framework and show how
performance is affected by incorporating various geometric and physics based
constraints, occluding vision sensors, or injecting noise in tactile sensors.
We also compare with prior work on multiple datasets and demonstrate that our
approach can effectively optimize over multi-modal sensor data and reduce
uncertainty to find better state estimates.
