  Recommendation strategies are typically evaluated by using previously logged
data, employing off-policy evaluation methods to estimate their expected
performance. However, for strategies that present users with slates of multiple
items, the resulting combinatorial action space renders many of these methods
impractical. Prior work has developed estimators that leverage the structure in
slates to estimate the expected off-policy performance, but the estimation of
the entire performance distribution remains elusive. Estimating the complete
distribution allows for a more comprehensive evaluation of recommendation
strategies, particularly along the axes of risk and fairness that employ
metrics computable from the distribution. In this paper, we propose an
estimator for the complete off-policy performance distribution for slates and
establish conditions under which the estimator is unbiased and consistent. This
builds upon prior work on off-policy evaluation for slates and off-policy
distribution estimation in reinforcement learning. We validate the efficacy of
our method empirically on synthetic data as well as on a slate recommendation
simulator constructed from real-world data (MovieLens-20M). Our results show a
significant reduction in estimation variance and improved sample efficiency
over prior work across a range of slate structures.
