  Multilingual information retrieval (MLIR) considers the problem of ranking
documents in several languages for a query expressed in a language that may
differ from any of those languages. Recent work has observed that approaches
such as combining ranked lists representing a single document language each or
using multilingual pretrained language models demonstrate a preference for one
language over others. This results in systematic unfair treatment of documents
in different languages. This work proposes a language fairness metric to
evaluate whether documents across different languages are fairly ranked through
statistical equivalence testing using the Kruskal-Wallis test. In contrast to
most prior work in group fairness, we do not consider any language to be an
unprotected group. Thus our proposed measure, PEER (Probability of
EqualExpected Rank), is the first fairness metric specifically designed to
capture the language fairness of MLIR systems. We demonstrate the behavior of
PEER on artificial ranked lists. We also evaluate real MLIR systems on two
publicly available benchmarks and show that the PEER scores align with prior
analytical findings on MLIR fairness. Our implementation is compatible with
ir-measures and is available at http://github.com/hltcoe/peer_measure.
