  Training energy-based models (EBMs) with noise-contrastive estimation (NCE)
is theoretically feasible but practically challenging. Effective learning
requires the noise distribution to be approximately similar to the target
distribution, especially in high-dimensional domains. Previous works have
explored modelling the noise distribution as a separate generative model, and
then concurrently training this noise model with the EBM. While this method
allows for more effective noise-contrastive estimation, it comes at the cost of
extra memory and training complexity. Instead, this thesis proposes a
self-adapting NCE algorithm which uses static instances of the EBM along its
training trajectory as the noise distribution. During training, these static
instances progressively converge to the target distribution, thereby
circumventing the need to simultaneously train an auxiliary noise model.
Moreover, we express this self-adapting NCE algorithm in the framework of
Bregman divergences and show that it is a generalization of maximum likelihood
learning for EBMs. The performance of our algorithm is evaluated across a range
of noise update intervals, and experimental results show that shorter update
intervals are conducive to higher synthesis quality.
