  There has been a recent explosion in research into machine-learning-based
generative modeling to tackle computational challenges for simulations in high
energy physics (HEP). In order to use such alternative simulators in practice,
we need well-defined metrics to compare different generative models and
evaluate their discrepancy from the true distributions. We present the first
systematic review and investigation into evaluation metrics and their
sensitivity to failure modes of generative models, using the framework of
two-sample goodness-of-fit testing, and their relevance and viability for HEP.
Inspired by previous work in both physics and computer vision, we propose two
new metrics, the Fr\'echet and kernel physics distances (FPD and KPD,
respectively), and perform a variety of experiments measuring their performance
on simple Gaussian-distributed, and simulated high energy jet datasets. We find
FPD, in particular, to be the most sensitive metric to all alternative jet
distributions tested and recommend its adoption, along with the KPD and
Wasserstein distances between individual feature distributions, for evaluating
generative models in HEP. We finally demonstrate the efficacy of these proposed
metrics in evaluating and comparing a novel attention-based generative
adversarial particle transformer to the state-of-the-art message-passing
generative adversarial network jet simulation model. The code for our proposed
metrics is provided in the open source JetNet Python library.
