  We present a method to learn a joint multimodal representation space that
enables recognition of unseen activities in videos. We first compare the effect
of placing various constraints on the embedding space using paired text and
video data. We also propose a method to improve the joint embedding space using
an adversarial formulation, allowing it to benefit from unpaired text and video
data. By using unpaired text data, we show the ability to learn a
representation that better captures unseen activities.
  In addition to testing on publicly available datasets, we introduce a new,
large-scale text/video dataset.
  We experimentally confirm that using paired and unpaired data to learn a
shared embedding space benefits three difficult tasks (i) zero-shot activity
classification, (ii) unsupervised activity discovery, and (iii) unseen activity
captioning, outperforming the state-of-the-arts.
