  We present CoNTACT: a Dutch language model adapted to the domain of COVID-19
tweets. The model was developed by continuing the pre-training phase of RobBERT
(Delobelle, 2020) by using 2.8M Dutch COVID-19 related tweets posted in 2021.
In order to test the performance of the model and compare it to RobBERT, the
two models were tested on two tasks: (1) binary vaccine hesitancy detection and
(2) detection of arguments for vaccine hesitancy. For both tasks, not only
Twitter but also Facebook data was used to show cross-genre performance. In our
experiments, CoNTACT showed statistically significant gains over RobBERT in all
experiments for task 1. For task 2, we observed substantial improvements in
virtually all classes in all experiments. An error analysis indicated that the
domain adaptation yielded better representations of domain-specific
terminology, causing CoNTACT to make more accurate classification decisions.
