  Diffusion probabilistic models (DPMs) are a class of powerful deep generative
models (DGMs). Despite their success, the iterative generation process over the
full timesteps is much less efficient than other DGMs such as GANs. Thus, the
generation performance on a subset of timesteps is crucial, which is greatly
influenced by the covariance design in DPMs. In this work, we consider diagonal
and full covariances to improve the expressive power of DPMs. We derive the
optimal result for such covariances, and then correct it when the mean of DPMs
is imperfect. Both the optimal and the corrected ones can be decomposed into
terms of conditional expectations over functions of noise. Building upon it, we
propose to estimate the optimal covariance and its correction given imperfect
mean by learning these conditional expectations. Our method can be applied to
DPMs with both discrete and continuous timesteps. We consider the diagonal
covariance in our implementation for computational efficiency. For an efficient
practical implementation, we adopt a parameter sharing scheme and a two-stage
training process. Empirically, our method outperforms a wide variety of
covariance design on likelihood results, and improves the sample quality
especially on a small number of timesteps.
