  Sequence-to-sequence models with soft attention have been successfully
applied to a wide variety of problems, but their decoding process incurs a
quadratic time and space cost and is inapplicable to real-time sequence
transduction. To address these issues, we propose Monotonic Chunkwise Attention
(MoChA), which adaptively splits the input sequence into small chunks over
which soft attention is computed. We show that models utilizing MoChA can be
trained efficiently with standard backpropagation while allowing online and
linear-time decoding at test time. When applied to online speech recognition,
we obtain state-of-the-art results and match the performance of a model using
an offline soft attention mechanism. In document summarization experiments
where we do not expect monotonic alignments, we show significantly improved
performance compared to a baseline monotonic attention-based model.
