  Reinforcement learning (RL) has shown great promise with algorithms learning
in environments with large state and action spaces purely from scalar reward
signals. A crucial challenge for current deep RL algorithms is that they
require a tremendous amount of environment interactions for learning. This can
be infeasible in situations where such interactions are expensive; such as in
robotics. Offline RL algorithms try to address this issue by bootstrapping the
learning process from existing logged data without needing to interact with the
environment from the very beginning. While online RL algorithms are typically
evaluated as a function of the number of environment interactions, there exists
no single established protocol for evaluating offline RL methods.In this paper,
we propose a sequential approach to evaluate offline RL algorithms as a
function of the training set size and thus by their data efficiency. Sequential
evaluation provides valuable insights into the data efficiency of the learning
process and the robustness of algorithms to distribution changes in the dataset
while also harmonizing the visualization of the offline and online learning
phases. Our approach is generally applicable and easy to implement. We compare
several existing offline RL algorithms using this approach and present insights
from a variety of tasks and offline datasets.
