  For large-scale IT corpora with hundreds of classes organized in a hierarchy,
the task of accurate classification of classes at the higher level in the
hierarchies is crucial to avoid errors propagating to the lower levels. In the
business world, an efficient and explainable ML model is preferred over an
expensive black-box model, especially if the performance increase is marginal.
A current trend in the Natural Language Processing (NLP) community is towards
employing huge pre-trained language models (PLMs) or what is known as
self-attention models (e.g., BERT) for almost any kind of NLP task (e.g.,
question-answering, sentiment analysis, text classification). Despite the
widespread use of PLMs and the impressive performance in a broad range of NLP
tasks, there is a lack of a clear and well-justified need to as why these
models are being employed for domain-specific text classification (TC) tasks,
given the monosemic nature of specialized words (i.e., jargon) found in
domain-specific text which renders the purpose of contextualized embeddings
(e.g., PLMs) futile. In this paper, we compare the accuracies of some
state-of-the-art (SOTA) models reported in the literature against a Linear SVM
classifier and TFIDF vectorization model on three TC datasets. Results show a
comparable performance for the LinearSVM. The findings of this study show that
for domain-specific TC tasks, a linear model can provide a comparable, cheap,
reproducible, and interpretable alternative to attention-based models.
