  Learning from unstructured and uncurated data has become the dominant
paradigm for generative approaches in language and vision. Such unstructured
and unguided behavior data, commonly known as play, is also easier to collect
in robotics but much more difficult to learn from due to its inherently
multimodal, noisy, and suboptimal nature. In this paper, we study this problem
of learning goal-directed skill policies from unstructured play data which is
labeled with language in hindsight. Specifically, we leverage advances in
diffusion models to learn a multi-task diffusion model to extract robotic
skills from play data. Using a conditional denoising diffusion process in the
space of states and actions, we can gracefully handle the complexity and
multimodality of play data and generate diverse and interesting robot
behaviors. To make diffusion models more useful for skill learning, we
encourage robotic agents to acquire a vocabulary of skills by introducing
discrete bottlenecks into the conditional behavior generation process. In our
experiments, we demonstrate the effectiveness of our approach across a wide
variety of environments in both simulation and the real world. Results
visualizations and videos at https://play-fusion.github.io
