  A number of variational autoencoders (VAEs) have recently emerged with the
aim of modeling multimodal data, e.g., to jointly model images and their
corresponding captions. Still, multimodal VAEs tend to focus solely on a subset
of the modalities, e.g., by fitting the image while neglecting the caption. We
refer to this limitation as modality collapse. In this work, we argue that this
effect is a consequence of conflicting gradients during multimodal VAE
training. We show how to detect the sub-graphs in the computational graphs
where gradients conflict (impartiality blocks), as well as how to leverage
existing gradient-conflict solutions from multitask learning to mitigate
modality collapse. That is, to ensure impartial optimization across modalities.
We apply our training framework to several multimodal VAE models, losses and
datasets from the literature, and empirically show that our framework
significantly improves the reconstruction performance, conditional generation,
and coherence of the latent space across modalities.
