  Pivot-based neural representation models have lead to significant progress in
domain adaptation for NLP. However, previous works that follow this approach
utilize only labeled data from the source domain and unlabeled data from the
source and target domains, but neglect to incorporate massive unlabeled corpora
that are not necessarily drawn from these domains. To alleviate this, we
propose PERL: A representation learning model that extends contextualized word
embedding models such as BERT with pivot-based fine-tuning. PERL outperforms
strong baselines across 22 sentiment classification domain adaptation setups,
improves in-domain model performance, yields effective reduced-size models and
increases model stability.
