  Despite the tremendous progress in zero-shot learning(ZSL), the majority of
existing methods still rely on human-annotated attributes, which are difficult
to annotate and scale. An unsupervised alternative is to represent each class
using the word embedding associated with its semantic class name. However, word
embeddings extracted from pre-trained language models do not necessarily
capture visual similarities, resulting in poor zero-shot performance. In this
work, we argue that online textual documents, e.g., Wikipedia, contain rich
visual descriptions about object classes, therefore can be used as powerful
unsupervised side information for ZSL. To this end, we propose I2DFormer, a
novel transformer-based ZSL framework that jointly learns to encode images and
documents by aligning both modalities in a shared embedding space. In order to
distill discriminative visual words from noisy documents, we introduce a new
cross-modal attention module that learns fine-grained interactions between
image patches and document words. Consequently, our I2DFormer not only learns
highly discriminative document embeddings that capture visual similarities but
also gains the ability to localize visually relevant words in image regions.
Quantitatively, we demonstrate that our I2DFormer significantly outperforms
previous unsupervised semantic embeddings under both zero-shot and generalized
zero-shot learning settings on three public datasets. Qualitatively, we show
that our method leads to highly interpretable results where document words can
be grounded in the image regions.
