  Robotic fabric manipulation has applications in home robotics, textiles,
senior care and surgery. Existing fabric manipulation techniques, however, are
designed for specific tasks, making it difficult to generalize across different
but related tasks. We extend the Visual Foresight framework to learn fabric
dynamics that can be efficiently reused to accomplish different fabric
manipulation tasks with a single goal-conditioned policy. We introduce
VisuoSpatial Foresight (VSF), which builds on prior work by learning visual
dynamics on domain randomized RGB images and depth maps simultaneously and
completely in simulation. We experimentally evaluate VSF on multi-step fabric
smoothing and folding tasks against 5 baseline methods in simulation and on the
da Vinci Research Kit (dVRK) surgical robot without any demonstrations at train
or test time. Furthermore, we find that leveraging depth significantly improves
performance. RGBD data yields an 80% improvement in fabric folding success rate
over pure RGB data. Code, data, videos, and supplementary material are
available at https://sites.google.com/view/fabric-vsf/.
