  Recent Automatic Speech Recognition systems have been moving towards
end-to-end systems that can be trained together. Numerous techniques that have
been proposed recently enabled this trend, including feature extraction with
CNNs, context capturing and acoustic feature modeling with RNNs, automatic
alignment of input and output sequences using Connectionist Temporal
Classifications, as well as replacing traditional n-gram language models with
RNN Language Models. Historically, there has been a lot of interest in
automatic punctuation in textual or speech to text context. However, there
seems to be little interest in incorporating automatic punctuation into the
emerging neural network based end-to-end speech recognition systems, partially
due to the lack of English speech corpus with punctuated transcripts. In this
study, we propose a method to generate punctuated transcript for the TEDLIUM
dataset using transcripts available from ted.com. We also propose an end-to-end
ASR system that outputs words and punctuations concurrently from speech
signals. Combining Damerau Levenshtein Distance and slot error rate into
DLev-SER, we enable measurement of punctuation error rate when the hypothesis
text is not perfectly aligned with the reference. Compared with previous
methods, our model reduces slot error rate from 0.497 to 0.341.
