  Deep reinforcement learning (RL) algorithms typically parameterize the policy
as a deep network that outputs either a deterministic action or a stochastic
one modeled as a Gaussian distribution, hence restricting learning to a single
behavioral mode. Meanwhile, diffusion models emerged as a powerful framework
for multimodal learning. However, the use of diffusion policies in online RL is
hindered by the intractability of policy likelihood approximation, as well as
the greedy objective of RL methods that can easily skew the policy to a single
mode. This paper presents Deep Diffusion Policy Gradient (DDiffPG), a novel
actor-critic algorithm that learns from scratch multimodal policies
parameterized as diffusion models while discovering and maintaining versatile
behaviors. DDiffPG explores and discovers multiple modes through off-the-shelf
unsupervised clustering combined with novelty-based intrinsic motivation.
DDiffPG forms a multimodal training batch and utilizes mode-specific Q-learning
to mitigate the inherent greediness of the RL objective, ensuring the
improvement of the diffusion policy across all modes. Our approach further
allows the policy to be conditioned on mode-specific embeddings to explicitly
control the learned modes. Empirical studies validate DDiffPG's capability to
master multimodal behaviors in complex, high-dimensional continuous control
tasks with sparse rewards, also showcasing proof-of-concept dynamic online
replanning when navigating mazes with unseen obstacles.
