  Natural language processing (NLP) algorithms are rapidly improving but often
struggle when applied to out-of-distribution examples. A prominent approach to
mitigate the domain gap is domain adaptation, where a model trained on a source
domain is adapted to a new target domain. We present a new learning setup,
``domain adaptation from scratch'', which we believe to be crucial for
extending the reach of NLP to sensitive domains in a privacy-preserving manner.
In this setup, we aim to efficiently annotate data from a set of source domains
such that the trained model performs well on a sensitive target domain from
which data is unavailable for annotation. Our study compares several approaches
for this challenging setup, ranging from data selection and domain adaptation
algorithms to active learning paradigms, on two NLP tasks: sentiment analysis
and Named Entity Recognition. Our results suggest that using the abovementioned
approaches eases the domain gap, and combining them further improves the
results.
