  Subwords are the most widely used output units in end-to-end speech
recognition. They combine the best of two worlds by modeling the majority of
frequent words directly and at the same time allow open vocabulary speech
recognition by backing off to shorter units or characters to construct words
unseen during training. However, mapping text to subwords is ambiguous and
often multiple segmentation variants are possible. Yet, many systems are
trained using only the most likely segmentation. Recent research suggests that
sampling subword segmentations during training acts as a regularizer for neural
machine translation and speech recognition models, leading to performance
improvements. In this work, we conduct a principled investigation on the
regularizing effect of the subword segmentation sampling method for a streaming
end-to-end speech recognition task. In particular, we evaluate the subword
regularization contribution depending on the size of the training dataset. Our
results suggest that subword regularization provides a consistent improvement
of (2-8%) relative word-error-rate reduction, even in a large-scale setting
with datasets up to a size of 20k hours. Further, we analyze the effect of
subword regularization on recognition of unseen words and its implications on
beam diversity.
