  Many manipulation tasks require the robot to control the contact between a
grasped compliant tool and the environment, e.g. scraping a frying pan with a
spatula. However, modeling tool-environment interaction is difficult,
especially when the tool is compliant, and the robot cannot be expected to have
the full geometry and physical properties (e.g., mass, stiffness, and friction)
of all the tools it must use. We propose a framework that learns to predict the
effects of a robot's actions on the contact between the tool and the
environment given visuo-tactile perception. Key to our framework is a novel
contact feature representation that consists of a binary contact value, the
line of contact, and an end-effector wrench. We propose a method to learn the
dynamics of these contact features from real world data that does not require
predicting the geometry of the compliant tool. We then propose a controller
that uses this dynamics model for visuo-tactile contact servoing and show that
it is effective at performing scraping tasks with a spatula, even in scenarios
where precise contact needs to be made to avoid obstacles.
