  Recent advances in batch (offline) reinforcement learning have shown
promising results in learning from available offline data and proved offline
reinforcement learning to be an essential toolkit in learning control policies
in a model-free setting. An offline reinforcement learning algorithm applied to
a dataset collected by a suboptimal non-learning-based algorithm can result in
a policy that outperforms the behavior agent used to collect the data. Such a
scenario is frequent in robotics, where existing automation is collecting
operational data. Although offline learning techniques can learn from data
generated by a sub-optimal behavior agent, there is still an opportunity to
improve the sample complexity of existing offline reinforcement learning
algorithms by strategically introducing human demonstration data into the
training process. To this end, we propose a novel approach that uses
uncertainty estimation to trigger the injection of human demonstration data and
guide policy training towards optimal behavior while reducing overall sample
complexity. Our experiments show that this approach is more sample efficient
when compared to a naive way of combining expert data with data collected from
a sub-optimal agent. We augmented an existing offline reinforcement learning
algorithm Conservative Q-Learning with our approach and performed experiments
on data collected from MuJoCo and OffWorld Gym learning environments.
