  Reinforcement learning in partially observable domains is challenging due to
the lack of observable state information. Thankfully, learning offline in a
simulator with such state information is often possible. In particular, we
propose a method for partially observable reinforcement learning that uses a
fully observable policy (which we call a state expert) during offline training
to improve online performance. Based on Soft Actor-Critic (SAC), our agent
balances performing actions similar to the state expert and getting high
returns under partial observability. Our approach can leverage the
fully-observable policy for exploration and parts of the domain that are fully
observable while still being able to learn under partial observability. On six
robotics domains, our method outperforms pure imitation, pure reinforcement
learning, the sequential or parallel combination of both types, and a recent
state-of-the-art method in the same setting. A successful policy transfer to a
physical robot in a manipulation task from pixels shows our approach's
practicality in learning interesting policies under partial observability.
