  Synthetic images generated from deep generative models have the potential to
address data scarcity and data privacy issues. The selection of synthesis
models is mostly based on image quality measurements, and most researchers
favor synthetic images that produce realistic images, i.e., images with good
fidelity scores, such as low Fr\'echet Inception Distance (FID) and high Peak
Signal-To-Noise Ratio (PSNR). However, the quality of synthetic images is not
limited to fidelity, and a wide spectrum of metrics should be evaluated to
comprehensively measure the quality of synthetic images. In addition, quality
metrics are not truthful predictors of the utility of synthetic images, and the
relations between these evaluation metrics are not yet clear. In this work, we
have established a comprehensive set of evaluators for synthetic images,
including fidelity, variety, privacy, and utility. By analyzing more than 100k
chest X-ray images and their synthetic copies, we have demonstrated that there
is an inevitable trade-off between synthetic image fidelity, variety, and
privacy. In addition, we have empirically demonstrated that the utility score
does not require images with both high fidelity and high variety. For intra-
and cross-task data augmentation, mode-collapsed images and low-fidelity images
can still demonstrate high utility. Finally, our experiments have also showed
that it is possible to produce images with both high utility and privacy, which
can provide a strong rationale for the use of deep generative models in
privacy-preserving applications. Our study can shore up comprehensive guidance
for the evaluation of synthetic images and elicit further developments for
utility-aware deep generative models in medical image synthesis.
