  Rank-Biased Overlap (RBO) is a similarity measure for indefinite rankings: it
is top-weighted, and can be computed when only a prefix of the rankings is
known or when they have only some items in common. It is widely used for
instance to analyze differences between search engines by comparing the
rankings of documents they retrieve for the same queries. In these situations,
though, it is very frequent to find tied documents that have the same score.
Unfortunately, the treatment of ties in RBO remains superficial and incomplete,
in the sense that it is not clear how to calculate it from the ranking prefixes
only. In addition, the existing way of dealing with ties is very different from
the one traditionally followed in the field of Statistics, most notably found
in rank correlation coefficients such as Kendall's and Spearman's. In this
paper we propose a generalized formulation for RBO to handle ties, thanks to
which we complete the original definitions by showing how to perform prefix
evaluation. We also use it to fully develop two variants that align with the
ones found in the Statistics literature: one when there is a reference ranking
to compare to, and one when there is not. Overall, these three variants provide
researchers with flexibility when comparing rankings with RBO, by clearly
determining what ties mean, and how they should be treated. Finally, using both
synthetic and TREC data, we demonstrate the use of these new tie-aware RBO
measures. We show that the scores may differ substantially from the original
tie-unaware RBO measure, where ties had to be broken at random or by arbitrary
criteria such as by document ID. Overall, these results evidence the need for a
proper account of ties in rank similarity measures such as RBO.
