  We introduce the concept of \emph{expected exposure} as the average attention
ranked items receive from users over repeated samples of the same query.
Furthermore, we advocate for the adoption of the principle of equal expected
exposure: given a fixed information need, no item should receive more or less
expected exposure than any other item of the same relevance grade. We argue
that this principle is desirable for many retrieval objectives and scenarios,
including topical diversity and fair ranking. Leveraging user models from
existing retrieval metrics, we propose a general evaluation methodology based
on expected exposure and draw connections to related metrics in information
retrieval evaluation. Importantly, this methodology relaxes classic information
retrieval assumptions, allowing a system, in response to a query, to produce a
\emph{distribution over rankings} instead of a single fixed ranking. We study
the behavior of the expected exposure metric and stochastic rankers across a
variety of information access conditions, including \emph{ad hoc} retrieval and
recommendation. We believe that measuring and optimizing expected exposure
metrics using randomization opens a new area for retrieval algorithm
development and progress.
