  Transformer-based models have demonstrated their effectiveness in automatic
speech recognition (ASR) tasks and even shown superior performance over the
conventional hybrid framework. The main idea of Transformers is to capture the
long-range global context within an utterance by self-attention layers.
However, for scenarios like conversational speech, such utterance-level
modeling will neglect contextual dependencies that span across utterances. In
this paper, we propose to explicitly model the inter-sentential information in
a Transformer based end-to-end architecture for conversational speech
recognition. Specifically, for the encoder network, we capture the contexts of
previous speech and incorporate such historic information into current input by
a context-aware residual attention mechanism. For the decoder, the prediction
of current utterance is also conditioned on the historic linguistic information
through a conditional decoder framework. We show the effectiveness of our
proposed method on several open-source dialogue corpora and the proposed method
consistently improved the performance from the utterance-level
Transformer-based ASR models.
