  Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) is a challenging cross-modal
retrieval task. In prior arts, the retrieval is conducted by sorting the
distance between the query sketch and each image in the gallery. However, the
domain gap and the zero-shot setting make neural networks hard to generalize.
This paper tackles the challenges from a new perspective: utilizing gallery
image features. We propose a Cluster-then-Retrieve (ClusterRetri) method that
performs clustering on the gallery images and uses the cluster centroids as
proxies for retrieval. Furthermore, a distribution alignment loss is proposed
to align the image and sketch features with a common Gaussian distribution,
reducing the domain gap. Despite its simplicity, our proposed method
outperforms the state-of-the-art methods by a large margin on popular datasets,
e.g., up to 31% and 39% relative improvement of mAP@all on the Sketchy and
TU-Berlin datasets.
