  Planning contact interactions is one of the core challenges of many robotic
tasks. Optimizing contact locations while taking dynamics into account is
computationally costly and, in environments that are only partially observable,
executing contact-based tasks often suffers from low accuracy. We present an
approach that addresses these two challenges for the problem of vision-based
manipulation. First, we propose to disentangle contact from motion
optimization. Thereby, we improve planning efficiency by focusing computation
on promising contact locations. Second, we use a hybrid approach for perception
and state estimation that combines neural networks with a physically meaningful
state representation. In simulation and real-world experiments on the task of
planar pushing, we show that our method is more efficient and achieves a higher
manipulation accuracy than previous vision-based approaches.
