  Energy-based latent variable models (EBLVMs) are more expressive than
conventional energy-based models. However, its potential on visual tasks are
limited by its training process based on maximum likelihood estimate that
requires sampling from two intractable distributions. In this paper, we propose
Bi-level doubly variational learning (BiDVL), which is based on a new bi-level
optimization framework and two tractable variational distributions to
facilitate learning EBLVMs. Particularly, we lead a decoupled EBLVM consisting
of a marginal energy-based distribution and a structural posterior to handle
the difficulties when learning deep EBLVMs on images. By choosing a symmetric
KL divergence in the lower level of our framework, a compact BiDVL for visual
tasks can be obtained. Our model achieves impressive image generation
performance over related works. It also demonstrates the significant capacity
of testing image reconstruction and out-of-distribution detection.
