  Recent advances in deep learning models for sequence classification have
greatly improved their classification accuracy, specially when large training
sets are available. However, several works have suggested that under some
settings the predictions made by these models are poorly calibrated. In this
work we study binary sequence classification problems and we look at model
calibration from a different perspective by asking the question: Are deep
learning models capable of learning the underlying target class distribution?
We focus on sparse sequence classification, that is problems in which the
target class is rare and compare three deep learning sequence classification
models. We develop an evaluation that measures how well a classifier is
learning the target class distribution. In addition, our evaluation
disentangles good performance achieved by mere compression of the training
sequences versus performance achieved by proper model generalization. Our
results suggest that in this binary setting the deep-learning models are indeed
able to learn the underlying class distribution in a non-trivial manner, i.e.
by proper generalization beyond data compression.
