  We present multimodal conditioning modules (MCM) for enabling conditional
image synthesis using pretrained diffusion models. Previous multimodal
synthesis works rely on training networks from scratch or fine-tuning
pretrained networks, both of which are computationally expensive for large,
state-of-the-art diffusion models. Our method uses pretrained networks but
\textit{does not require any updates to the diffusion network's parameters}.
MCM is a small module trained to modulate the diffusion network's predictions
during sampling using 2D modalities (e.g., semantic segmentation maps,
sketches) that were unseen during the original training of the diffusion model.
We show that MCM enables user control over the spatial layout of the image and
leads to increased control over the image generation process. Training MCM is
cheap as it does not require gradients from the original diffusion net,
consists of only $\sim$1$\%$ of the number of parameters of the base diffusion
model, and is trained using only a limited number of training examples. We
evaluate our method on unconditional and text-conditional models to demonstrate
the improved control over the generated images and their alignment with respect
to the conditioning inputs.
