  Recent breakthroughs in scale have enabled the emergence of powerful
generative language models, and the ability to fine-tune these models on
various tasks by casting them into prompts or instructions. In this landscape,
the problem of Unsupervised Domain Adaptation (UDA), or the problem of
leveraging knowledge from a labeled source domain to an unlabeled target
domain, has been left behind, with recent UDA methods still addressing
discriminative classification. In particular, two popular UDA approaches,
involving Continued Pre-Training (CPT) and learning domain invariant
representations, have been under-explored in the generative setting, signaling
a gap. In this work, we evaluate the utility of CPT for generative UDA. We
first perform an empirical evaluation to measure the trade-offs between CPT and
strong methods promoting domain invariance. We further evaluate how well the
benefits of CPT extend to different architectures, tuning methods and data
regimes. We then motivate the use of CPT by studying to what degree it benefits
classification performance on the target domain. Finally, we attempt to
understand the mechanism behind which CPT improves classification performance
on the unlabeled target domain. Our findings suggest that a implicitly learns
the downstream task while predicting masked words informative to that task. Our
work connects the body of UDA research with that of instruction tuning,
enabling an initial step towards a wider applicability of modern language
models.
