  Imitation learning addresses the challenge of learning by observing an
expert's demonstrations without access to reward signals from environments.
Most existing imitation learning methods that do not require interacting with
environments either model the expert distribution as the conditional
probability p(a|s) (e.g., behavioral cloning, BC) or the joint probability p(s,
a). Despite the simplicity of modeling the conditional probability with BC, it
usually struggles with generalization. While modeling the joint probability can
improve generalization performance, the inference procedure is often
time-consuming, and the model can suffer from manifold overfitting. This work
proposes an imitation learning framework that benefits from modeling both the
conditional and joint probability of the expert distribution. Our proposed
Diffusion Model-Augmented Behavioral Cloning (DBC) employs a diffusion model
trained to model expert behaviors and learns a policy to optimize both the BC
loss (conditional) and our proposed diffusion model loss (joint). DBC
outperforms baselines in various continuous control tasks in navigation, robot
arm manipulation, dexterous manipulation, and locomotion. We design additional
experiments to verify the limitations of modeling either the conditional
probability or the joint probability of the expert distribution, as well as
compare different generative models. Ablation studies justify the effectiveness
of our design choices.
