  In this work, we describe a novel method of training an embedding-matching
word-level connectionist temporal classification (CTC) automatic speech
recognizer (ASR) such that it directly produces word start times and durations,
required by many real-world applications, in addition to the transcription. The
word timestamps enable the ASR to output word segmentations and word confusion
networks without relying on a secondary model or forced alignment process when
testing. Our proposed system has similar word segmentation accuracy as a hybrid
DNN-HMM (Deep Neural Network-Hidden Markov Model) system, with less than 3ms
difference in mean absolute error in word start times on TIMIT data. At the
same time, we observed less than 5% relative increase in the word error rate
compared to the non-timestamped system when using the same audio training data
and nearly identical model size. We also contribute more rigorous analysis of
multiple-hypothesis embedding-matching ASR in general.
