  This work combines information about the dialogue history encoded by
pre-trained model with a meaning representation of the current system utterance
to realize contextual language generation in task-oriented dialogues. We
utilize the pre-trained multi-context ConveRT model for context representation
in a model trained from scratch; and leverage the immediate preceding user
utterance for context generation in a model adapted from the pre-trained GPT-2.
Both experiments with the MultiWOZ dataset show that contextual information
encoded by pre-trained model improves the performance of response generation
both in automatic metrics and human evaluation. Our presented contextual
generator enables higher variety of generated responses that fit better to the
ongoing dialogue. Analysing the context size shows that longer context does not
automatically lead to better performance, but the immediate preceding user
utterance plays an essential role for contextual generation. In addition, we
also propose a re-ranker for the GPT-based generation model. The experiments
show that the response selected by the re-ranker has a significant improvement
on automatic metrics.
