  Ranking metrics are a family of metrics largely used to evaluate recommender
systems. However they typically suffer from the fact the reward is affected by
the order in which recommended items are displayed to the user. A classical way
to overcome this position bias is to uniformly shuffle a proportion of the
recommendations, but this method may result in a bad user experience. It is
nevertheless common to use a stochastic policy to generate the recommendations,
and we suggest a new method to overcome the position bias, by leveraging the
stochasticity of the policy used to collect the dataset.
