  Conversational context information, higher-level knowledge that spans across
sentences, can help to recognize a long conversation. However, existing speech
recognition models are typically built at a sentence level, and thus it may not
capture important conversational context information. The recent progress in
end-to-end speech recognition enables integrating context with other available
information (e.g., acoustic, linguistic resources) and directly recognizing
words from speech. In this work, we present a direct acoustic-to-word,
end-to-end speech recognition model capable of utilizing the conversational
context to better process long conversations. We evaluate our proposed approach
on the Switchboard conversational speech corpus and show that our system
outperforms a standard end-to-end speech recognition system.
