  Learning robotic tasks in the real world is still highly challenging and
effective practical solutions remain to be found. Traditional methods used in
this area are imitation learning and reinforcement learning, but they both have
limitations when applied to real robots. Combining reinforcement learning with
pre-collected demonstrations is a promising approach that can help in learning
control policies to solve robotic tasks. In this paper, we propose an algorithm
that uses novel techniques to leverage offline expert data using offline and
online training to obtain faster convergence and improved performance. The
proposed algorithm (AWET) weights the critic losses with a novel agent
advantage weight to improve over the expert data. In addition, AWET makes use
of an automatic early termination technique to stop and discard policy rollouts
that are not similar to expert trajectories -- to prevent drifting far from the
expert data. In an ablation study, AWET showed improved and promising
performance when compared to state-of-the-art baselines on four standard
robotic tasks.
