  The task of multimodal learning has seen a growing interest recently as it
allows for training neural architectures based on different modalities such as
vision, text, and audio. One challenge in training such models is that they
need to jointly learn semantic concepts and their relationships across
different input representations. Capsule networks have been shown to perform
well in context of capturing the relation between low-level input features and
higher-level concepts. However, capsules have so far mainly been used only in
small-scale fully supervised settings due to the resource demand of
conventional routing algorithms. We present a new multimodal capsule network
that allows us to leverage the strength of capsules in the context of a
multimodal learning framework on large amounts of video data. To adapt the
capsules to large-scale input data, we propose a novel routing by
self-attention mechanism that selects relevant capsules which are then used to
generate a final joint multimodal feature representation. This allows not only
for robust training with noisy video data, but also to scale up the size of the
capsule network compared to traditional routing methods while still being
computationally efficient. We evaluate the proposed architecture by pretraining
it on a large-scale multimodal video dataset and applying it on four datasets
in two challenging downstream tasks. Results show that the proposed multimodal
capsule network is not only able to improve results compared to other routing
techniques, but also achieves competitive performance on the task of multimodal
learning.
