  Pre-trained Transformers are challenging human performances in many NLP
tasks. The massive datasets used for pre-training seem to be the key to their
success on existing tasks. In this paper, we explore how a range of pre-trained
Natural Language Understanding models perform on definitely unseen sentences
provided by classification tasks over a DarkNet corpus. Surprisingly, results
show that syntactic and lexical neural networks perform on par with pre-trained
Transformers even after fine-tuning. Only after what we call extreme domain
adaptation, that is, retraining with the masked language model task on all the
novel corpus, pre-trained Transformers reach their standard high results. This
suggests that huge pre-training corpora may give Transformers unexpected help
since they are exposed to many of the possible sentences.
