  Off-policy evaluation (OPE) in contextual bandits has seen rapid adoption in
real-world systems, since it enables offline evaluation of new policies using
only historic log data. Unfortunately, when the number of actions is large,
existing OPE estimators -- most of which are based on inverse propensity score
weighting -- degrade severely and can suffer from extreme bias and variance.
This foils the use of OPE in many applications from recommender systems to
language models. To overcome this issue, we propose a new OPE estimator that
leverages marginalized importance weights when action embeddings provide
structure in the action space. We characterize the bias, variance, and mean
squared error of the proposed estimator and analyze the conditions under which
the action embedding provides statistical benefits over conventional
estimators. In addition to the theoretical analysis, we find that the empirical
performance improvement can be substantial, enabling reliable OPE even when
existing estimators collapse due to a large number of actions.
