  With the increasing research interest in dialogue response generation, there
is an emerging branch formulating this task as selecting next sentences, where
given the partial dialogue contexts, the goal is to determine the most probable
next sentence. Following the recent success of the Transformer model, this
paper proposes (1) a new variant of attention mechanism based on multi-head
attention, called highway attention, and (2) a recurrent model based on
transformer and the proposed highway attention, so-called Highway Recurrent
Transformer. Experiments on the response selection task in the seventh Dialog
System Technology Challenge (DSTC7) show the capability of the proposed model
of modeling both utterance-level and dialogue-level information; the
effectiveness of each module is further analyzed as well.
