  This study evaluates the robustness of two state-of-the-art deep contextual
language representations, ELMo and DistilBERT, on supervised learning of binary
protest news classification and sentiment analysis of product reviews. A
"cross-context" setting is enabled using test sets that are distinct from the
training data. Specifically, in the news classification task, the models are
developed on local news from India and tested on the local news from China. In
the sentiment analysis task, the models are trained on movie reviews and tested
on customer reviews. This comparison is aimed at exploring the limits of the
representative power of today's Natural Language Processing systems on the path
to the systems that are generalizable to real-life scenarios. The models are
fine-tuned and fed into a Feed-Forward Neural Network and a Bidirectional Long
Short Term Memory network. Multinomial Naive Bayes and Linear Support Vector
Machine are used as traditional baselines. The results show that, in binary
text classification, DistilBERT is significantly better than ELMo on
generalizing to the cross-context setting. ELMo is observed to be significantly
more robust to the cross-context test data than both baselines. On the other
hand, the baselines performed comparably well to ELMo when the training and
test data are subsets of the same corpus (no cross-context). DistilBERT is also
found to be 30% smaller and 83% faster than ELMo. The results suggest that
DistilBERT can transfer generic semantic knowledge to other domains better than
ELMo. DistilBERT is also favorable in incorporating into real-life systems for
it requires a smaller computational training budget. When generalization is not
the utmost preference and test domain is similar to the training domain, the
traditional ML algorithms can still be considered as more economic alternatives
to deep language representations.
