  In this paper, we introduce a novel family of iterative algorithms which
carry out $\alpha$-divergence minimisation in a Variational Inference context.
They do so by ensuring a systematic decrease at each step in the
$\alpha$-divergence between the variational and the posterior distributions. In
its most general form, the variational distribution is a mixture model and our
framework allows us to simultaneously optimise the weights and components
parameters of this mixture model. Our approach permits us to build on various
methods previously proposed for $\alpha$-divergence minimisation such as
Gradient or Power Descent schemes and we also shed a new light on an integrated
Expectation Maximization algorithm. Lastly, we provide empirical evidence that
our methodology yields improved results on several multimodal target
distributions and on a real data example.
