  Deep generative models parametrized up to a normalizing constant (e.g.
energy-based models) are difficult to train by maximizing the likelihood of the
data because the likelihood and/or gradients thereof cannot be explicitly or
efficiently written down. Score matching is a training method, whereby instead
of fitting the likelihood $\log p(x)$ for the training data, we instead fit the
score function $\nabla_x \log p(x)$ -- obviating the need to evaluate the
partition function. Though this estimator is known to be consistent, its
unclear whether (and when) its statistical efficiency is comparable to that of
maximum likelihood -- which is known to be (asymptotically) optimal. We
initiate this line of inquiry in this paper, and show a tight connection
between statistical efficiency of score matching and the isoperimetric
properties of the distribution being estimated -- i.e. the Poincar\'e,
log-Sobolev and isoperimetric constant -- quantities which govern the mixing
time of Markov processes like Langevin dynamics. Roughly, we show that the
score matching estimator is statistically comparable to the maximum likelihood
when the distribution has a small isoperimetric constant. Conversely, if the
distribution has a large isoperimetric constant -- even for simple families of
distributions like exponential families with rich enough sufficient statistics
-- score matching will be substantially less efficient than maximum likelihood.
We suitably formalize these results both in the finite sample regime, and in
the asymptotic regime. Finally, we identify a direct parallel in the discrete
setting, where we connect the statistical properties of pseudolikelihood
estimation with approximate tensorization of entropy and the Glauber dynamics.
