  Deep reinforcement learning (DRL) provides a new way to generate robot
control policy. However, the process of training control policy requires
lengthy exploration, resulting in a low sample efficiency of reinforcement
learning (RL) in real-world tasks. Both imitation learning (IL) and learning
from demonstrations (LfD) improve the training process by using expert
demonstrations, but imperfect expert demonstrations can mislead policy
improvement. Offline to Online reinforcement learning requires a lot of offline
data to initialize the policy, and distribution shift can easily lead to
performance degradation during online fine-tuning. To solve the above problems,
we propose a learning from demonstrations method named A-SILfD, which treats
expert demonstrations as the agent's successful experiences and uses
experiences to constrain policy improvement. Furthermore, we prevent
performance degradation due to large estimation errors in the Q-function by the
ensemble Q-functions. Our experiments show that A-SILfD can significantly
improve sample efficiency using a small number of different quality expert
demonstrations. In four Mujoco continuous control tasks, A-SILfD can
significantly outperform baseline methods after 150,000 steps of online
training and is not misled by imperfect expert demonstrations during training.
