  In this paper, we show that citation counts work better than a random
baseline (by a margin of 10%) in distinguishing excellent research, while
Mendeley reader counts don't work better than the baseline. Specifically, we
study the potential of these metrics for distinguishing publications that
caused a change in a research field from those that have not. The experiment
has been conducted on a new dataset for bibliometric research called
TrueImpactDataset. TrueImpactDataset is a collection of research publications
of two types -- research papers which are considered seminal works in their
area and papers which provide a literature review of a research area. We
provide overview statistics of the dataset and propose to use it for validating
research evaluation metrics. Using the dataset, we conduct a set of experiments
to study how citation and reader counts perform in distinguishing these
publication types, following the intuition that causing a change in a field
signifies research contribution. We show that citation counts help in
distinguishing research that strongly influenced later developments from works
that predominantly discuss the current state of the art with a degree of
accuracy (63%, i.e. 10% over the random baseline). In all setups, Mendeley
reader counts perform worse than a random baseline.
