  Human intervention is an effective way to inject human knowledge into the
training loop of reinforcement learning, which can bring fast learning and
ensured training safety. Given the very limited budget of human intervention,
it remains challenging to design when and how human expert interacts with the
learning agent in the training. In this work, we develop a novel
human-in-the-loop learning method called Human-AI Copilot Optimization
(HACO).To allow the agent's sufficient exploration in the risky environments
while ensuring the training safety, the human expert can take over the control
and demonstrate how to avoid probably dangerous situations or trivial
behaviors. The proposed HACO then effectively utilizes the data both from the
trial-and-error exploration and human's partial demonstration to train a
high-performing agent. HACO extracts proxy state-action values from partial
human demonstration and optimizes the agent to improve the proxy values
meanwhile reduce the human interventions. The experiments show that HACO
achieves a substantially high sample efficiency in the safe driving benchmark.
HACO can train agents to drive in unseen traffic scenarios with a handful of
human intervention budget and achieve high safety and generalizability,
outperforming both reinforcement learning and imitation learning baselines with
a large margin. Code and demo videos are available at:
https://decisionforce.github.io/HACO/.
