  Remarkable gains in deep learning usually rely on tremendous supervised data.
Ensuring the modality diversity for one object in training set is critical for
the generalization of cutting-edge deep models, but it burdens human with heavy
manual labor on data collection and annotation. In addition, some rare or
unexpected modalities are new for the current model, causing reduced
performance under such emerging modalities. Inspired by the achievements in
speech recognition, psychology and behavioristics, we present a practical
solution, self-reinforcing unsupervised matching (SUM), to annotate the images
with 2D structure-preserving property in an emerging modality by cross-modality
matching. This approach requires no any supervision in emerging modality and
only one template in seen modality, providing a possible route towards
continual learning.
