  Zero-shot sketch-based image retrieval typically asks for a trained model to
be applied as is to unseen categories. In this paper, we question to argue that
this setup by definition is not compatible with the inherent abstract and
subjective nature of sketches, i.e., the model might transfer well to new
categories, but will not understand sketches existing in different test-time
distribution as a result. We thus extend ZS-SBIR asking it to transfer to both
categories and sketch distributions. Our key contribution is a test-time
training paradigm that can adapt using just one sketch. Since there is no
paired photo, we make use of a sketch raster-vector reconstruction module as a
self-supervised auxiliary task. To maintain the fidelity of the trained
cross-modal joint embedding during test-time update, we design a novel
meta-learning based training paradigm to learn a separation between model
updates incurred by this auxiliary task from those off the primary objective of
discriminative learning. Extensive experiments show our model to outperform
state of-the-arts, thanks to the proposed test-time adaption that not only
transfers to new categories but also accommodates to new sketching styles.
