  In end-to-end automatic speech recognition (ASR), a model is expected to
implicitly learn representations suitable for recognizing a word-level
sequence. However, the huge abstraction gap between input acoustic signals and
output linguistic tokens makes it challenging for a model to learn the
representations. In this work, to promote the word-level representation
learning in end-to-end ASR, we propose a hierarchical conditional model that is
based on connectionist temporal classification (CTC). Our model is trained by
auxiliary CTC losses applied to intermediate layers, where the vocabulary size
of each target subword sequence is gradually increased as the layer becomes
close to the word-level output. Here, we make each level of sequence prediction
explicitly conditioned on the previous sequences predicted at lower levels.
With the proposed approach, we expect the proposed model to learn the
word-level representations effectively by exploiting a hierarchy of linguistic
structures. Experimental results on LibriSpeech-{100h, 960h} and TEDLIUM2
demonstrate that the proposed model improves over a standard CTC-based model
and other competitive models from prior work. We further analyze the results to
confirm the effectiveness of the intended representation learning with our
model.
