  In this paper, we ask the research question of whether all the datasets in
the benchmark are necessary. We approach this by first characterizing the
distinguishability of datasets when comparing different systems. Experiments on
9 datasets and 36 systems show that several existing benchmark datasets
contribute little to discriminating top-scoring systems, while those less used
datasets exhibit impressive discriminative power. We further, taking the text
classification task as a case study, investigate the possibility of predicting
dataset discrimination based on its properties (e.g., average sentence length).
Our preliminary experiments promisingly show that given a sufficient number of
training experimental records, a meaningful predictor can be learned to
estimate dataset discrimination over unseen datasets. We released all datasets
with features explored in this work on DataLab:
\url{https://datalab.nlpedia.ai}.
