  The emergence of pre-trained language models (PLMs) has shown great success
in many Natural Language Processing (NLP) tasks including text classification.
Due to the minimal to no feature engineering required when using these models,
PLMs are becoming the de facto choice for any NLP task. However, for
domain-specific corpora (e.g., financial, legal, and industrial), fine-tuning a
pre-trained model for a specific task has shown to provide a performance
improvement. In this paper, we compare the performance of four different PLMs
on three public domain-free datasets and a real-world dataset containing
domain-specific words, against a simple SVM linear classifier with TFIDF
vectorized text. The experimental results on the four datasets show that using
PLMs, even fine-tuned, do not provide significant gain over the linear SVM
classifier. Hence, we recommend that for text classification tasks, traditional
SVM along with careful feature engineering can pro-vide a cheaper and superior
performance than PLMs.
