  In reinforcement learning (RL) research, it is common to assume access to
direct online interactions with the environment. However in many real-world
applications, access to the environment is limited to a fixed offline dataset
of logged experience. In such settings, standard RL algorithms have been shown
to diverge or otherwise yield poor performance. Accordingly, recent work has
suggested a number of remedies to these issues. In this work, we introduce a
general framework, behavior regularized actor critic (BRAC), to empirically
evaluate recently proposed methods as well as a number of simple baselines
across a variety of offline continuous control tasks. Surprisingly, we find
that many of the technical complexities introduced in recent methods are
unnecessary to achieve strong performance. Additional ablations provide
insights into which design choices matter most in the offline RL setting.
