  Many recent developments on generative models for natural images have relied
on heuristically-motivated metrics that can be easily gamed by memorizing a
small sample from the true distribution or training a model directly to improve
the metric. In this work, we critically evaluate the gameability of these
metrics by designing and deploying a generative modeling competition. Our
competition received over 11000 submitted models. The competitiveness between
participants allowed us to investigate both intentional and unintentional
memorization in generative modeling. To detect intentional memorization, we
propose the ``Memorization-Informed Fr\'echet Inception Distance'' (MiFID) as a
new memorization-aware metric and design benchmark procedures to ensure that
winning submissions made genuine improvements in perceptual quality.
Furthermore, we manually inspect the code for the 1000 top-performing models to
understand and label different forms of memorization. Our analysis reveals that
unintentional memorization is a serious and common issue in popular generative
models. The generated images and our memorization labels of those models as
well as code to compute MiFID are released to facilitate future studies on
benchmarking generative models.
