  Generalising deep networks to novel domains without manual labels is
challenging to deep learning. This problem is intrinsically difficult due to
unpredictable changing nature of imagery data distributions in novel domains.
Pre-learned knowledge does not transfer well without making strong assumptions
about the learned and the novel domains. Different methods have been studied to
address the underlying problem based on different assumptions, e.g. from domain
adaptation to zero-shot and few-shot learning. In this work, we address this
problem by transfer clustering that aims to learn a discriminative latent space
of the unlabelled target data in a novel domain by knowledge transfer from
labelled related domains. Specifically, we want to leverage relative (pairwise)
imagery information, which is freely available and intrinsic to a target
domain, to model the target domain image distribution characteristics as well
as the prior-knowledge learned from related labelled domains to enable more
discriminative clustering of unlabelled target data. Our method mitigates
nontransferrable prior-knowledge by self-supervision, benefiting from both
transfer and self-supervised learning. Extensive experiments on four datasets
for image clustering tasks reveal the superiority of our model over the
state-of-the-art transfer clustering techniques. We further demonstrate its
competitive transferability on four zero-shot learning benchmarks.
