  Heterogeneous data modalities can provide complementary cues for several
tasks, usually leading to more robust algorithms and better performance.
However, while training data can be accurately collected to include a variety
of sensory modalities, it is often the case that not all of them are available
in real life (testing) scenarios, where a model has to be deployed. This raises
the challenge of how to extract information from multimodal data in the
training stage, in a form that can be exploited at test time, considering
limitations such as noisy or missing modalities. This paper presents a new
approach in this direction for RGB-D vision tasks, developed within the
adversarial learning and privileged information frameworks. We consider the
practical case of learning representations from depth and RGB videos, while
relying only on RGB data at test time. We propose a new approach to train a
hallucination network that learns to distill depth information via adversarial
learning, resulting in a clean approach without several losses to balance or
hyperparameters. We report state-of-the-art results on object classification on
the NYUD dataset and video action recognition on the largest multimodal dataset
available for this task, the NTU RGB+D, as well as on the Northwestern-UCLA.
