  Deformable object manipulation presents a unique set of challenges in robotic
manipulation by exhibiting high degrees of freedom and severe self-occlusion.
State representation for materials that exhibit plastic behavior, like modeling
clay or bread dough, is also difficult because they permanently deform under
stress and are constantly changing shape. In this work, we investigate each of
these challenges using the task of robotic sculpting with a parallel gripper.
We propose a system that uses point clouds as the state representation and
leverages pre-trained point cloud reconstruction Transformer to learn a latent
dynamics model to predict material deformations given a grasp action. We design
a novel action sampling algorithm that reasons about geometrical differences
between point clouds to further improve the efficiency of model-based planners.
All data and experiments are conducted entirely in the real world. Our
experiments show the proposed system is able to successfully capture the
dynamics of clay, and is able to create a variety of simple shapes.
