  Zero shot learning (ZSL) has seen a surge in interest over the decade for its
tight links with the mechanism making young children recognize novel objects.
Although different paradigms of visual semantic embedding models are designed
to align visual features and distributed word representations, it is unclear to
what extent current ZSL models encode semantic information from distributed
word representations. In this work, we introduce the split of tiered-ImageNet
to the ZSL task, in order to avoid the structural flaws in the standard
ImageNet benchmark. We build a unified framework for ZSL with contrastive
learning as pre-training, which guarantees no semantic information leakage and
encourages linearly separable visual features. Our work makes it fair for
evaluating visual semantic embedding models on a ZSL setting in which semantic
inference is decisive. With this framework, we show that current ZSL models
struggle with encoding semantic relationships from word analogy and word
hierarchy. Our analyses provide motivation for exploring the role of context
language representations in ZSL tasks.
