  Modern decision-making systems, from robots to web recommendation engines,
are expected to adapt: to user preferences, changing circumstances or even new
tasks. Yet, it is still uncommon to deploy a dynamically learning agent (rather
than a fixed policy) to a production system, as it's perceived as unsafe. Using
historical data to reason about learning algorithms, similar to offline policy
evaluation (OPE) applied to fixed policies, could help practitioners evaluate
and ultimately deploy such adaptive agents to production. In this work, we
formalize offline learner simulation (OLS) for reinforcement learning (RL) and
propose a novel evaluation protocol that measures both fidelity and efficiency
of the simulation. For environments with complex high-dimensional observations,
we propose a semi-parametric approach that leverages recent advances in latent
state discovery in order to achieve accurate and efficient offline simulations.
In preliminary experiments, we show the advantage of our approach compared to
fully non-parametric baselines. The code to reproduce these experiments will be
made available at https://github.com/microsoft/rl-offline-simulation.
