  Measuring the generalization capacity of Deep Generative Models (DGMs) is
difficult because of the curse of dimensionality. Evaluation metrics for DGMs
such as Inception Score, Fr\'echet Inception Distance, Precision-Recall, and
Neural Net Divergence try to estimate the distance between the generated
distribution and the target distribution using a polynomial number of samples.
These metrics are the target of researchers when designing new models. Despite
the claims, it is still unclear how well can they measure the generalization
capacity of a generative model. In this paper, we investigate the capacity of
these metrics in measuring the generalization capacity. We introduce a
framework for comparing the robustness of evaluation metrics. We show that
better scores in these metrics do not imply better generalization. They can be
fooled easily by a generator that memorizes a small subset of the training set.
We propose a fix to the NND metric to make it more robust to noise in the
generated data. Toward building a robust metric for generalization, we propose
to apply the Minimum Description Length principle to the problem of evaluating
DGMs. We develop an efficient method for estimating the complexity of
Generative Latent Variable Models (GLVMs). Experimental results show that our
metric can effectively detect training set memorization and distinguish GLVMs
of different generalization capacities. Source code is available at
https://github.com/htt210/GeneralizationMetricGAN.
