  The variational lower bound (a.k.a. ELBO or free energy) is the central
objective for many established as well as many novel algorithms for
unsupervised learning. During learning such algorithms change model parameters
to increase the variational lower bound. Learning usually proceeds until
parameters have converged to values close to a stationary point of the learning
dynamics. In this purely theoretical contribution, we show that (for a very
large class of generative models) the variational lower bound is at all
stationary points of learning equal to a sum of entropies. For standard machine
learning models with one set of latents and one set of observed variables, the
sum consists of three entropies: (A) the (average) entropy of the variational
distributions, (B) the negative entropy of the model's prior distribution, and
(C) the (expected) negative entropy of the observable distribution. The
obtained result applies under realistic conditions including: finite numbers of
data points, at any stationary point (including saddle points) and for any
family of (well behaved) variational distributions. The class of generative
models for which we show the equality to entropy sums contains many well-known
generative models. As concrete examples we discuss Sigmoid Belief Networks,
probabilistic PCA and (Gaussian and non-Gaussian) mixture models. The result
also applies for standard (Gaussian) variational autoencoders, a special case
that has been shown previously (Damm et al., 2023). The prerequisites we use to
show equality to entropy sums are relatively mild. Concretely, the
distributions of a given generative model have to be of the exponential family,
and the model has to satisfy a parameterization criterion (which is usually
fulfilled). Proving the equality of the ELBO to entropy sums at stationary
points (under the stated conditions) is the main contribution of this work.
