  We introduce NIFT, Neural Interaction Field and Template, a descriptive and
robust interaction representation of object manipulations to facilitate
imitation learning. Given a few object manipulation demos, NIFT guides the
generation of the interaction imitation for a new object instance by matching
the Neural Interaction Template (NIT) extracted from the demos in the target
Neural Interaction Field (NIF) defined for the new object. Specifically, the
NIF is a neural field that encodes the relationship between each spatial point
and a given object, where the relative position is defined by a spherical
distance function rather than occupancies or signed distances, which are
commonly adopted by conventional neural fields but less informative. For a
given demo interaction, the corresponding NIT is defined by a set of spatial
points sampled in the demo NIF with associated neural features. To better
capture the interaction, the points are sampled on the Interaction Bisector
Surface (IBS), which consists of points that are equidistant to the two
interacting objects and has been used extensively for interaction
representation. With both point selection and pointwise features defined for
better interaction encoding, NIT effectively guides the feature matching in the
NIFs of the new object instances such that the relative poses are optimized to
realize the manipulation while imitating the demo interactions. Experiments
show that our NIFT solution outperforms state-of-the-art imitation learning
methods for object manipulation and generalizes better to objects from new
categories.
