  The ability to discover optimal behaviour from fixed data sets has the
potential to transfer the successes of reinforcement learning (RL) to domains
where data collection is acutely problematic. In this offline setting, a key
challenge is overcoming overestimation bias for actions not present in data
which, without the ability to correct for via interaction with the environment,
can propagate and compound during training, leading to highly sub-optimal
policies. One simple method to reduce this bias is to introduce a policy
constraint via behavioural cloning (BC), which encourages agents to pick
actions closer to the source data. By finding the right balance between RL and
BC such approaches have been shown to be surprisingly effective while requiring
minimal changes to the underlying algorithms they are based on. To date this
balance has been held constant, but in this work we explore the idea of tipping
this balance towards RL following initial training. Using TD3-BC, we
demonstrate that by continuing to train a policy offline while reducing the
influence of the BC component we can produce refined policies that outperform
the original baseline, as well as match or exceed the performance of more
complex alternatives. Furthermore, we demonstrate such an approach can be used
for stable online fine-tuning, allowing policies to be safely improved during
deployment.
