  The Listen, Attend and Spell (LAS) model and other attention-based automatic
speech recognition (ASR) models have known limitations when operated in a fully
online mode. In this paper, we analyze the online operation of LAS models to
demonstrate that these limitations stem from the handling of silence regions
and the reliability of online attention mechanism at the edge of input buffers.
We propose a novel and simple technique that can achieve fully online
recognition while meeting accuracy and latency targets. For the Mandarin
dictation task, our proposed approach can achieve a character error rate in
online operation that is within 4% relative to an offline LAS model. The
proposed online LAS model operates at 12% lower latency relative to a
conventional neural network hidden Markov model hybrid of comparable accuracy.
We have validated the proposed method through a production scale deployment,
which, to the best of our knowledge, is the first such deployment of a fully
online LAS model.
