  Large-scale pre-trained language models such as BERT are popular solutions
for text classification. Due to the superior performance of these advanced
methods, nowadays, people often directly train them for a few epochs and deploy
the obtained model. In this opinion paper, we point out that this way may only
sometimes get satisfactory results. We argue the importance of running a simple
baseline like linear classifiers on bag-of-words features along with advanced
methods. First, for many text data, linear methods show competitive
performance, high efficiency, and robustness. Second, advanced models such as
BERT may only achieve the best results if properly applied. Simple baselines
help to confirm whether the results of advanced models are acceptable. Our
experimental results fully support these points.
