  Many sequential decision making problems are high-stakes and require
off-policy evaluation (OPE) of a new policy using historical data collected
using some other policy. One of the most common OPE techniques that provides
unbiased estimates is trajectory based importance sampling (IS). However, due
to the high variance of trajectory IS estimates, importance sampling methods
based on state-action visitation distributions (SIS) have recently been
adopted. Unfortunately, while SIS often provides lower variance estimates for
long horizons, estimating the state-action distribution ratios can be
challenging and lead to biased estimates. In this paper, we present a new
perspective on this bias-variance trade-off and show the existence of a
spectrum of estimators whose endpoints are SIS and IS. Additionally, we also
establish a spectrum for doubly-robust and weighted version of these
estimators. We provide empirical evidence that estimators in this spectrum can
be used to trade-off between the bias and variance of IS and SIS and can
achieve lower mean-squared error than both IS and SIS.
