  Text classification is a fundamental Natural Language Processing task that
has a wide variety of applications, where deep learning approaches have
produced state-of-the-art results. While these models have been heavily
criticized for their black-box nature, their robustness to slight perturbations
in input text has been a matter of concern. In this work, we carry out a
data-focused study evaluating the impact of systematic practical perturbations
on the performance of the deep learning based text classification models like
CNN, LSTM, and BERT-based algorithms. The perturbations are induced by the
addition and removal of unwanted tokens like punctuation and stop-words that
are minimally associated with the final performance of the model. We show that
these deep learning approaches including BERT are sensitive to such legitimate
input perturbations on four standard benchmark datasets SST2, TREC-6, BBC News,
and tweet_eval. We observe that BERT is more susceptible to the removal of
tokens as compared to the addition of tokens. Moreover, LSTM is slightly more
sensitive to input perturbations as compared to CNN based model. The work also
serves as a practical guide to assessing the impact of discrepancies in
train-test conditions on the final performance of models.
