  Cross-domain text classification aims to adapt models to a target domain that
lacks labeled data. It leverages or reuses rich labeled data from the different
but related source domain(s) and unlabeled data from the target domain. To this
end, previous work focuses on either extracting domain-invariant features or
task-agnostic features, ignoring domain-aware features that may be present in
the target domain and could be useful for the downstream task. In this paper,
we propose a two-stage framework for cross-domain text classification. In the
first stage, we finetune the model with mask language modeling (MLM) and
labeled data from the source domain. In the second stage, we further fine-tune
the model with self-supervised distillation (SSD) and unlabeled data from the
target domain. We evaluate its performance on a public cross-domain text
classification benchmark and the experiment results show that our method
achieves new state-of-the-art results for both single-source domain adaptations
(94.17% $\uparrow$1.03%) and multi-source domain adaptations (95.09%
$\uparrow$1.34%).
