  Recurrent neural networks are deep learning topologies that can be trained to
classify long documents. However, in our recent work, we found a critical
problem with these cells: they can use the length differences between texts of
different classes as a prominent classification feature. This has the effect of
producing models that are brittle and fragile to concept drift, can provide
misleading performances and are trivially explainable regardless of text
content. This paper illustrates the problem using synthetic and real-world data
and provides a simple solution using weight decay regularization.
