  We address the challenge of policy evaluation in real-world applications of
reinforcement learning systems where the available historical data is limited
due to ethical, practical, or security considerations. This constrained
distribution of data samples often leads to biased policy evaluation estimates.
To remedy this, we propose that instead of policy evaluation, one should
perform policy comparison, i.e. to rank the policies of interest in terms of
their value based on available historical data. In addition we present the
Limited Data Estimator (LDE) as a simple method for evaluating and comparing
policies from a small number of interactions with the environment. According to
our theoretical analysis, the LDE is shown to be statistically reliable on
policy comparison tasks under mild assumptions on the distribution of the
historical data. Additionally, our numerical experiments compare the LDE to
other policy evaluation methods on the task of policy ranking and demonstrate
its advantage in various settings.
