  The comparison of alternative rankings of a set of items is a general and
prominent task in applied statistics. Predictor variables are ranked according
to magnitude of association with an outcome, prediction models rank subjects
according to the personalized risk of an event, and genetic studies rank genes
according to their difference in gene expression levels. This article
constructs measures of the agreement of two or more ordered lists. We use the
standard deviation of the ranks to define a measure of agreement that both
provides an intuitive interpretation and can be applied to any number of lists
even if some or all are incomplete or censored. The approach can identify
change-points in the agreement of the lists and the sequential changes of
agreement as a function of the depth of the lists can be compared graphically
to a permutation based reference set. The usefulness of these tools are
illustrated using gene rankings, and using data from two Danish ovarian cancer
studies where we assess the within and between agreement of different
statistical classification methods.
