  Even though Variational Autoencoders (VAEs) are widely used for
semi-supervised learning, the reason why they work remains unclear. In fact,
the addition of the unsupervised objective is most often vaguely described as a
regularization. The strength of this regularization is controlled by
down-weighting the objective on the unlabeled part of the training set. Through
an analysis of the objective of semi-supervised VAEs, we observe that they use
the posterior of the learned generative model to guide the inference model in
learning the partially observed latent variable. We show that given this
observation, it is possible to gain finer control on the effect of the
unsupervised objective on the training procedure. Using importance weighting,
we derive two novel objectives that prioritize either one of the partially
observed latent variable, or the unobserved latent variable. Experiments on the
IMDB english sentiment analysis dataset and on the AG News topic classification
dataset show the improvements brought by our prioritization mechanism and
exhibit a behavior that is inline with our description of the inner working of
Semi-Supervised VAEs.
