  Objects rarely sit in isolation in human environments. As such, we'd like our
robots to reason about how multiple objects relate to one another and how those
relations may change as the robot interacts with the world. To this end, we
propose a novel graph neural network framework for multi-object manipulation to
predict how inter-object relations change given robot actions. Our model
operates on partial-view point clouds and can reason about multiple objects
dynamically interacting during the manipulation. By learning a dynamics model
in a learned latent graph embedding space, our model enables multi-step
planning to reach target goal relations. We show our model trained purely in
simulation transfers well to the real world. Our planner enables the robot to
rearrange a variable number of objects with a range of shapes and sizes using
both push and pick and place skills.
