  Diffusion policies are conditional diffusion models that learn robot action
distributions conditioned on the robot and environment state. They have
recently shown to outperform both deterministic and alternative action
distribution learning formulations. 3D robot policies use 3D scene feature
representations aggregated from a single or multiple camera views using sensed
depth. They have shown to generalize better than their 2D counterparts across
camera viewpoints. We unify these two lines of work and present 3D Diffuser
Actor, a neural policy equipped with a novel 3D denoising transformer that
fuses information from the 3D visual scene, a language instruction and
proprioception to predict the noise in noised 3D robot pose trajectories. 3D
Diffuser Actor sets a new state-of-the-art on RLBench with an absolute
performance gain of 18.1% over the current SOTA on a multi-view setup and an
absolute gain of 13.1% on a single-view setup. On the CALVIN benchmark, it
improves over the current SOTA by a 9% relative increase. It also learns to
control a robot manipulator in the real world from a handful of demonstrations.
Through thorough comparisons with the current SOTA policies and ablations of
our model, we show 3D Diffuser Actor's design choices dramatically outperform
2D representations, regression and classification objectives, absolute
attentions, and holistic non-tokenized 3D scene embeddings.
