  Sensor fusion can significantly improve the performance of many computer
vision tasks. However, traditional fusion approaches are either not data-driven
and cannot exploit prior knowledge nor find regularities in a given dataset or
they are restricted to a single application. We overcome this shortcoming by
presenting a novel deep hierarchical variational autoencoder called FusionVAE
that can serve as a basis for many fusion tasks. Our approach is able to
generate diverse image samples that are conditioned on multiple noisy,
occluded, or only partially visible input images. We derive and optimize a
variational lower bound for the conditional log-likelihood of FusionVAE. In
order to assess the fusion capabilities of our model thoroughly, we created
three novel datasets for image fusion based on popular computer vision
datasets. In our experiments, we show that FusionVAE learns a representation of
aggregated information that is relevant to fusion tasks. The results
demonstrate that our approach outperforms traditional methods significantly.
Furthermore, we present the advantages and disadvantages of different design
choices.
