  We propose a framework that learns a representation transferable across
different domains and tasks in a label efficient manner. Our approach battles
domain shift with a domain adversarial loss, and generalizes the embedding to
novel task using a metric learning-based approach. Our model is simultaneously
optimized on labeled source data and unlabeled or sparsely labeled data in the
target domain. Our method shows compelling results on novel classes within a
new domain even when only a few labeled examples per class are available,
outperforming the prevalent fine-tuning approach. In addition, we demonstrate
the effectiveness of our framework on the transfer learning task from image
object recognition to video action recognition.
