  We introduce Compartmentalized Diffusion Models (CDM), a method to train
different diffusion models (or prompts) on distinct data sources and
arbitrarily compose them at inference time. The individual models can be
trained in isolation, at different times, and on different distributions and
domains and can be later composed to achieve performance comparable to a
paragon model trained on all data simultaneously. Furthermore, each model only
contains information about the subset of the data it was exposed to during
training, enabling several forms of training data protection. In particular,
CDMs enable perfect selective forgetting and continual learning for large-scale
diffusion models, allow serving customized models based on the user's access
rights. Empirically the quality (FID) of the class-conditional CDMs (8-splits)
is within 10% (on fine-grained vision datasets) of a monolithic model (no
splits), and allows (8x) faster forgetting compared monolithic model with a
maximum FID increase of 1%. When applied to text-to-image generation, CDMs
improve alignment (TIFA) by 14.33% over a monolithic model trained on MSCOCO.
CDMs also allow determining the importance of a subset of the data
(attribution) in generating particular samples, and reduce memorization.
