  Autoregressive models used to generate responses in open-domain dialogue
systems often struggle to take long-term context into account and to maintain
consistency over a dialogue. Previous research in open-domain dialogue
generation has shown that the use of \emph{auxiliary tasks} can introduce
inductive biases that encourage the model to improve these qualities. However,
most previous research has focused on encoder-only or encoder/decoder models,
while the use of auxiliary tasks in \emph{decoder-only} autoregressive models
is under-explored. This paper describes an investigation where four different
auxiliary tasks are added to small and medium-sized GPT-2 models fine-tuned on
the PersonaChat and DailyDialog datasets. The results show that the
introduction of the new auxiliary tasks leads to small but consistent
improvement in evaluations of the investigated models.
