  Diffusion-based generative modeling has been achieving state-of-the-art
results on various generation tasks. Most diffusion models, however, are
limited to a single-generation modeling. Can we generalize diffusion models
with the ability of multi-modal generative training for more generalizable
modeling? In this paper, we propose a principled way to define a diffusion
model by constructing a unified multi-modal diffusion model in a common
diffusion space. We define the forward diffusion process to be driven by an
information aggregation from multiple types of task-data, e.g., images for a
generation task and labels for a classification task. In the reverse process,
we enforce information sharing by parameterizing a shared backbone denoising
network with additional modality-specific decoder heads. Such a structure can
simultaneously learn to generate different types of multi-modal data with a
multi-task loss, which is derived from a new multi-modal variational lower
bound that generalizes the standard diffusion model. We propose several
multimodal generation settings to verify our framework, including image
transition, masked-image training, joint image-label and joint
image-representation generative modeling. Extensive experimental results on
ImageNet indicate the effectiveness of our framework for various multi-modal
generative modeling, which we believe is an important research direction worthy
of more future explorations.
