  We propose a learning problem involving adapting a pre-trained source model
to the target domain for classifying all classes that appeared in the source
data, using target data that covers only a partial label space. This problem is
practical, as it is unrealistic for the target end-users to collect data for
all classes prior to adaptation. However, it has received limited attention in
the literature. To shed light on this issue, we construct benchmark datasets
and conduct extensive experiments to uncover the inherent challenges. We found
a dilemma -- on the one hand, adapting to the new target domain is important to
claim better performance; on the other hand, we observe that preserving the
classification accuracy of classes missing in the target adaptation data is
highly challenging, let alone improving them. To tackle this, we identify two
key directions: 1) disentangling domain gradients from classification
gradients, and 2) preserving class relationships. We present several effective
solutions that maintain the accuracy of the missing classes and enhance the
overall performance, establishing solid baselines for holistic transfer of
pre-trained models with partial target data.
