  We address the problem of learning fine-grained cross-modal representations.
We propose an instance-based deep metric learning approach in joint visual and
textual space. The key novelty of this paper is that it shows that using
per-image semantic supervision leads to substantial improvement in zero-shot
performance over using class-only supervision. On top of that, we provide a
probabilistic justification for a metric rescaling approach that solves a very
common problem in the generalized zero-shot learning setting, i.e., classifying
test images from unseen classes as one of the classes seen during training. We
evaluate our approach on two fine-grained zero-shot learning datasets: CUB and
FLOWERS. We find that on the generalized zero-shot classification task CLAREL
consistently outperforms the existing approaches on both datasets.
