  In this paper, we consider the problem of fine-grained image retrieval in an
incremental setting, when new categories are added over time. On the one hand,
repeatedly training the representation on the extended dataset is
time-consuming. On the other hand, fine-tuning the learned representation only
with the new classes leads to catastrophic forgetting. To this end, we propose
an incremental learning method to mitigate retrieval performance degradation
caused by the forgetting issue. Without accessing any samples of the original
classes, the classifier of the original network provides soft "labels" to
transfer knowledge to train the adaptive network, so as to preserve the
previous capability for classification. More importantly, a regularization
function based on Maximum Mean Discrepancy is devised to minimize the
discrepancy of new classes features from the original network and the adaptive
network, respectively. Extensive experiments on two datasets show that our
method effectively mitigates the catastrophic forgetting on the original
classes while achieving high performance on the new classes.
