  This paper proposes a method to combine reinforcement learning (RL) and
imitation learning (IL) using a dynamic, performance-based modulation over
learning signals. The proposed method combines RL and behavioral cloning (IL),
or corrective feedback in the action space (interactive IL/IIL), by dynamically
weighting the losses to be optimized, taking into account the backpropagated
gradients used to update the policy and the agent's estimated performance. In
this manner, RL and IL/IIL losses are combined by equalizing their impact on
the policy's updates, while modulating said impact such that IL signals are
prioritized at the beginning of the learning process, and as the agent's
performance improves, the RL signals become progressively more relevant,
allowing for a smooth transition from pure IL/IIL to pure RL. The proposed
method is used to learn local planning policies for mobile robots, synthesizing
IL/IIL signals online by means of a scripted policy. An extensive evaluation of
the application of the proposed method to this task is performed in
simulations, and it is empirically shown that it outperforms pure RL in terms
of sample efficiency (achieving the same level of performance in the training
environment utilizing approximately 4 times less experiences), while
consistently producing local planning policies with better performance metrics
(achieving an average success rate of 0.959 in an evaluation environment,
outperforming pure RL by 12.5% and pure IL by 13.9%). Furthermore, the obtained
local planning policies are successfully deployed in the real world without
performing any major fine tuning. The proposed method can extend existing RL
algorithms, and is applicable to other problems for which generating IL/IIL
signals online is feasible. A video summarizing some of the real world
experiments that were conducted can be found in https://youtu.be/mZlaXn9WGzw.
