  Score matching (SM) provides a compelling approach to learn energy-based
models (EBMs) by avoiding the calculation of partition function. However, it
remains largely open to learn energy-based latent variable models (EBLVMs),
except some special cases. This paper presents a bi-level score matching (BiSM)
method to learn EBLVMs with general structures by reformulating SM as a
bi-level optimization problem. The higher level introduces a variational
posterior of the latent variables and optimizes a modified SM objective, and
the lower level optimizes the variational posterior to fit the true posterior.
To solve BiSM efficiently, we develop a stochastic optimization algorithm with
gradient unrolling. Theoretically, we analyze the consistency of BiSM and the
convergence of the stochastic algorithm. Empirically, we show the promise of
BiSM in Gaussian restricted Boltzmann machines and highly nonstructural EBLVMs
parameterized by deep convolutional neural networks. BiSM is comparable to the
widely adopted contrastive divergence and SM methods when they are applicable;
and can learn complex EBLVMs with intractable posteriors to generate natural
images.
