  As Diffusion Models have shown promising performance, a lot of efforts have
been made to improve the controllability of Diffusion Models. However, how to
train Diffusion Models to have the disentangled latent spaces and how to
naturally incorporate the disentangled conditions during the sampling process
have been underexplored. In this paper, we present a training framework for
feature disentanglement of Diffusion Models (FDiff). We further propose two
sampling methods that can boost the realism of our Diffusion Models and also
enhance the controllability. Concisely, we train Diffusion Models conditioned
on two latent features, a spatial content mask, and a flattened style
embedding. We rely on the inductive bias of the denoising process of Diffusion
Models to encode pose/layout information in the content feature and
semantic/style information in the style feature. Regarding the sampling
methods, we first generalize Composable Diffusion Models (GCDM) by breaking the
conditional independence assumption to allow for some dependence between
conditional inputs, which is shown to be effective in realistic generation in
our experiments. Second, we propose timestep-dependent weight scheduling for
content and style features to further improve the performance. We also observe
better controllability of our proposed methods compared to existing methods in
image manipulation and image translation.
