  Human demonstrations can provide trustful samples to train reinforcement
learning algorithms for robots to learn complex behaviors in real-world
environments. However, obtaining sufficient demonstrations may be impractical
because many behaviors are difficult for humans to demonstrate. A more
practical approach is to replace human demonstrations by human queries, i.e.,
preference-based reinforcement learning. One key limitation of the existing
algorithms is the need for a significant amount of human queries because a
large number of labeled data is needed to train neural networks for the
approximation of a continuous, high-dimensional reward function. To reduce and
minimize the need for human queries, we propose a new GAN-assisted human
preference-based reinforcement learning approach that uses a generative
adversarial network (GAN) to actively learn human preferences and then replace
the role of human in assigning preferences. The adversarial neural network is
simple and only has a binary output, hence requiring much less human queries to
train. Moreover, a maximum entropy based reinforcement learning algorithm is
designed to shape the loss towards the desired regions or away from the
undesired regions. To show the effectiveness of the proposed approach, we
present some studies on complex robotic tasks without access to the environment
reward in a typical MuJoCo robot locomotion environment. The obtained results
show our method can achieve a reduction of about 99.8% human time without
performance sacrifice.
