  Acoustic-to-Word recognition provides a straightforward solution to
end-to-end speech recognition without needing external decoding, language model
re-scoring or lexicon. While character-based models offer a natural solution to
the out-of-vocabulary problem, word models can be simpler to decode and may
also be able to directly recognize semantically meaningful units. We present
effective methods to train Sequence-to-Sequence models for direct word-level
recognition (and character-level recognition) and show an absolute improvement
of 4.4-5.0\% in Word Error Rate on the Switchboard corpus compared to prior
work. In addition to these promising results, word-based models are more
interpretable than character models, which have to be composed into words using
a separate decoding step. We analyze the encoder hidden states and the
attention behavior, and show that location-aware attention naturally represents
words as a single speech-word-vector, despite spanning multiple frames in the
input. We finally show that the Acoustic-to-Word model also learns to segment
speech into words with a mean standard deviation of 3 frames as compared with
human annotated forced-alignments for the Switchboard corpus.
