  Maintaining a consistent persona is a key quality for any open domain
dialogue system. Current state-of-the-art systems do this by training agents
with supervised learning or online reinforcement learning (RL). However,
systems trained with supervised learning often lack consistency as they are
never punished for uttering contradictions. Additional training with RL can
alleviate some of these issues, however the training process is expensive.
Instead, we propose an offline RL framework to improve the persona consistency
of dialogue systems. Our framework allows us to combine the advantages of
previous methods as we can inexpensively train our model on existing data as in
supervised learning, while punishing and rewarding specific utterances as in
RL. We also introduce a simple importance sampling method to reduce the
variance of importance weights in offline RL training which we call
Variance-Reducing MLE-Initialized (VaRMI) importance sampling. Our automatic
and human evaluations show that our framework improves both the persona
consistency and dialogue quality of a state-of-the-art social chatbot.
