  Probabilistic generative models provide a powerful framework for representing
data that avoids the expense of manual annotation typically needed by
discriminative approaches. Model selection in this generative setting can be
challenging, however, particularly when likelihoods are not easily accessible.
To address this issue, we introduce a statistical test of relative similarity,
which is used to determine which of two models generates samples that are
significantly closer to a real-world reference dataset of interest. We use as
our test statistic the difference in maximum mean discrepancies (MMDs) between
the reference dataset and each model dataset, and derive a powerful,
low-variance test based on the joint asymptotic distribution of the MMDs
between each reference-model pair. In experiments on deep generative models,
including the variational auto-encoder and generative moment matching network,
the tests provide a meaningful ranking of model performance as a function of
parameter and training settings.
