  We evaluated the effectiveness of using language models, that were
pre-trained in one domain, as the basis for a classification model in another
domain: Dutch book reviews. Pre-trained language models have opened up new
possibilities for classification tasks with limited labelled data, because
representation can be learned in an unsupervised fashion. In our experiments we
have studied the effects of training set size (100-1600 items) on the
prediction accuracy of a ULMFiT classifier, based on a language models that we
pre-trained on the Dutch Wikipedia. We also compared ULMFiT to Support Vector
Machines, which is traditionally considered suitable for small collections. We
found that ULMFiT outperforms SVM for all training set sizes and that
satisfactory results (~90%) can be achieved using training sets that can be
manually annotated within a few hours. We deliver both our new benchmark
collection of Dutch book reviews for sentiment classification as well as the
pre-trained Dutch language model to the community.
