  In this paper, we study the problem of zero-shot sketch-based image retrieval
(ZS-SBIR). The prior methods tackle the problem in a two-modality setting with
only category labels or even no textual information involved. However, the
growing prevalence of Large-scale pre-trained Language Models (LLMs), which
have demonstrated great knowledge learned from web-scale data, can provide us
with an opportunity to conclude collective textual information. Our key
innovation lies in the usage of text data as auxiliary information for images,
thus leveraging the inherent zero-shot generalization ability that language
offers. To this end, we propose an approach called Cross-Modal Attention
Alignment Network with Auxiliary Text Description for zero-shot sketch-based
image retrieval. The network consists of three components: (i) a Description
Generation Module that generates textual descriptions for each training
category by prompting an LLM with several interrogative sentences, (ii) a
Feature Extraction Module that includes two ViTs for sketch and image data, a
transformer for extracting tokens of sentences of each training category,
finally (iii) a Cross-modal Alignment Module that exchanges the token features
of both text-sketch and text-image using cross-attention mechanism, and align
the tokens locally and globally. Extensive experiments on three benchmark
datasets show our superior performances over the state-of-the-art ZS-SBIR
methods.
