  Image and text retrieval is one of the foundational tasks in the vision and
language domain with multiple real-world applications. State-of-the-art
approaches, e.g. CLIP, ALIGN, represent images and texts as dense embeddings
and calculate the similarity in the dense embedding space as the matching
score. On the other hand, sparse semantic features like bag-of-words models are
more interpretable, but believed to suffer from inferior accuracy than dense
representations. In this work, we show that it is possible to build a sparse
semantic representation that is as powerful as, or even better than, dense
presentations. We extend the CLIP model and build a sparse text and image
representation (STAIR), where the image and text are mapped to a sparse token
space. Each token in the space is a (sub-)word in the vocabulary, which is not
only interpretable but also easy to integrate with existing information
retrieval systems. STAIR model significantly outperforms a CLIP model with
+$4.9\%$ and +$4.3\%$ absolute Recall@1 improvement on COCO-5k
text$\rightarrow$image and image$\rightarrow$text retrieval respectively. It
also achieved better performance on both of ImageNet zero-shot and linear
probing compared to CLIP.
