  General-purpose object placement is a fundamental capability of an
intelligent generalist robot, i.e., being capable of rearranging objects
following human instructions even in novel environments. To achieve this, we
break the rearrangement down into three parts, including object localization,
goal imagination and robot control, and propose a framework named SPORT. SPORT
leverages pre-trained large vision models for broad semantic reasoning about
objects, and learns a diffusion-based 3D pose estimator to ensure
physically-realistic results. Only object types (to be moved or reference) are
communicated between these two parts, which brings two benefits. One is that we
can fully leverage the powerful ability of open-set object localization and
recognition since no specific fine-tuning is needed for robotic scenarios.
Furthermore, the diffusion-based estimator only need to "imagine" the poses of
the moving and reference objects after the placement, while no necessity for
their semantic information. Thus the training burden is greatly reduced and no
massive training is required. The training data for goal pose estimation is
collected in simulation and annotated with GPT-4. A set of simulation and
real-world experiments demonstrate the potential of our approach to accomplish
general-purpose object rearrangement, placing various objects following precise
instructions.
