  Expectation-Maximization (EM) is a prominent approach for parameter
estimation of hidden (aka latent) variable models. Given the full batch of
data, EM forms an upper-bound of the negative log-likelihood of the model at
each iteration and updates to the minimizer of this upper-bound. We first
provide a "model level" interpretation of the EM upper-bound as sum of relative
entropy divergences to a set of singleton models, induced by the set of
observations. Our alternative motivation unifies the "observation level" and
the "model level" view of the EM. As a result, we formulate an online version
of the EM algorithm by adding an analogous inertia term which corresponds to
the relative entropy divergence to the old model. Our motivation is more widely
applicable than the previous approaches and leads to simple online updates for
mixture of exponential distributions, hidden Markov models, and the first known
online update for Kalman filters. Additionally, the finite sample form of the
inertia term lets us derive online updates when there is no closed-form
solution. Finally, we extend the analysis to the distributed setting where we
motivate a systematic way of combining multiple hidden variable models.
Experimentally, we validate the results on synthetic as well as real-world
datasets.
