  In this paper, we establish a connection between the parameterization of
flow-based and energy-based generative models, and present a new flow-based
modeling approach called energy-based normalizing flow (EBFlow). We demonstrate
that by optimizing EBFlow with score-matching objectives, the computation of
Jacobian determinants for linear transformations can be entirely bypassed. This
feature enables the use of arbitrary linear layers in the construction of
flow-based models without increasing the computational time complexity of each
training iteration from $O(D^2L)$ to $O(D^3L)$ for an $L$-layered model that
accepts $D$-dimensional inputs. This makes the training of EBFlow more
efficient than the commonly-adopted maximum likelihood training method. In
addition to the reduction in runtime, we enhance the training stability and
empirical performance of EBFlow through a number of techniques developed based
on our analysis of the score-matching methods. The experimental results
demonstrate that our approach achieves a significant speedup compared to
maximum likelihood estimation while outperforming prior methods with a
noticeable margin in terms of negative log-likelihood (NLL).
