  Manipulation planning is the task of computing robot trajectories that move a
set of objects to their target configuration while satisfying physically
feasibility. In contrast to existing works that assume known object templates,
we are interested in manipulation planning for a category of objects with
potentially unknown instances and large intra-category shape variation. To
achieve it, we need an object representation with which the manipulation
planner can reason about both the physical feasibility and desired object
configuration, while being generalizable to novel instances. The widely-used
pose representation is not suitable, as representing an object with a
parameterized transformation from a fixed template cannot capture large
intra-category shape variation. Hence, we propose a new hybrid object
representation consisting of semantic keypoint and dense geometry (a point
cloud or mesh) as the interface between the perception module and motion
planner. Leveraging advances in learning-based keypoint detection and shape
completion, both dense geometry and keypoints can be perceived from raw sensor
input. Using the proposed hybrid object representation, we formulate the
manipulation task as a motion planning problem which encodes both the object
target configuration and physical feasibility for a category of objects. In
this way, many existing manipulation planners can be generalized to categories
of objects, and the resulting perception-to-action manipulation pipeline is
robust to large intra-category shape variation. Extensive hardware experiments
demonstrate our pipeline can produce robot trajectories that accomplish tasks
with never-before-seen objects.
