  This paper presents a novel Learning from Demonstration (LfD) method that
uses neural fields to learn new skills efficiently and accurately. It achieves
this by utilizing a shared embedding to learn both scene and motion
representations in a generative way. Our method smoothly maps each expert
demonstration to a scene-motion embedding and learns to model them without
requiring hand-crafted task parameters or large datasets. It achieves data
efficiency by enforcing scene and motion generation to be smooth with respect
to changes in the embedding space. At inference time, our method can retrieve
scene-motion embeddings using test time optimization, and generate precise
motion trajectories for novel scenes. The proposed method is versatile and can
employ images, 3D shapes, and any other scene representations that can be
modeled using neural fields. Additionally, it can generate both end-effector
positions and joint angle-based trajectories. Our method is evaluated on tasks
that require accurate motion trajectory generation, where the underlying task
parametrization is based on object positions and geometric scene changes.
Experimental results demonstrate that the proposed method outperforms the
baseline approaches and generalizes to novel scenes. Furthermore, in real-world
experiments, we show that our method can successfully model multi-valued
trajectories, it is robust to the distractor objects introduced at inference
time, and it can generate 6D motions.
