  End-to-end task bots are typically learned over a static and usually
limited-size corpus. However, when deployed in dynamic, changing, and open
environments to interact with users, task bots tend to fail when confronted
with data that deviate from the training corpus, i.e., out-of-distribution
samples. In this paper, we study the problem of automatically adapting task
bots to changing environments by learning from human-bot interactions with
minimum or zero human annotations. We propose SL-AGENT, a novel self-learning
framework for building end-to-end task bots. SL-AGENT consists of a dialog
model and a pre-trained reward model to predict the quality of an agent
response. It enables task bots to automatically adapt to changing environments
by learning from the unlabeled human-bot dialog logs accumulated after
deployment via reinforcement learning with the incorporated reward model.
Experimental results on four well-studied dialog tasks show the effectiveness
of SL-AGENT to automatically adapt to changing environments, using both
automatic and human evaluations. We will release code and data for further
research.
