  We present a method for performing tasks involving spatial relations between
novel object instances initialized in arbitrary poses directly from point cloud
observations. Our framework provides a scalable way for specifying new tasks
using only 5-10 demonstrations. Object rearrangement is formalized as the
question of finding actions that configure task-relevant parts of the object in
a desired alignment. This formalism is implemented in three steps: assigning a
consistent local coordinate frame to the task-relevant object parts,
determining the location and orientation of this coordinate frame on unseen
object instances, and executing an action that brings these frames into the
desired alignment. We overcome the key technical challenge of determining
task-relevant local coordinate frames from a few demonstrations by developing
an optimization method based on Neural Descriptor Fields (NDFs) and a single
annotated 3D keypoint. An energy-based learning scheme to model the joint
configuration of the objects that satisfies a desired relational task further
improves performance. The method is tested on three multi-object rearrangement
tasks in simulation and on a real robot. Project website, videos, and code:
https://anthonysimeonov.github.io/r-ndf/
