  This paper is the first work to propose a network to predict a structured
uncertainty distribution for a synthesized image. Previous approaches have been
mostly limited to predicting diagonal covariance matrices. Our novel model
learns to predict a full Gaussian covariance matrix for each reconstruction,
which permits efficient sampling and likelihood evaluation.
  We demonstrate that our model can accurately reconstruct ground truth
correlated residual distributions for synthetic datasets and generate plausible
high frequency samples for real face images. We also illustrate the use of
these predicted covariances for structure preserving image denoising.
