  We present a framework for solving long-horizon planning problems involving
manipulation of rigid objects that operates directly from a point-cloud
observation, i.e. without prior object models. Our method plans in the space of
object subgoals and frees the planner from reasoning about robot-object
interaction dynamics by relying on a set of generalizable manipulation
primitives. We show that for rigid bodies, this abstraction can be realized
using low-level manipulation skills that maintain sticking contact with the
object and represent subgoals as 3D transformations. To enable generalization
to unseen objects and improve planning performance, we propose a novel way of
representing subgoals for rigid-body manipulation and a graph-attention based
neural network architecture for processing point-cloud inputs. We
experimentally validate these choices using simulated and real-world
experiments on the YuMi robot. Results demonstrate that our method can
successfully manipulate new objects into target configurations requiring
long-term planning. Overall, our framework realizes the best of the worlds of
task-and-motion planning (TAMP) and learning-based approaches. Project website:
https://anthonysimeonov.github.io/rpo-planning-framework/.
