  Temporal connectionist temporal classification (CTC)-based automatic speech
recognition (ASR) is one of the most successful end to end (E2E) ASR
frameworks. However, due to the token independence assumption in decoding, an
external language model (LM) is required which destroys its fast parallel
decoding property. Several studies have been proposed to transfer linguistic
knowledge from a pretrained LM (PLM) to the CTC based ASR. Since the PLM is
built from text while the acoustic model is trained with speech, a cross-modal
alignment is required in order to transfer the context dependent linguistic
knowledge from the PLM to acoustic encoding. In this study, we propose a novel
cross-modal alignment algorithm based on optimal transport (OT). In the
alignment process, a transport coupling matrix is obtained using OT, which is
then utilized to transform a latent acoustic representation for matching the
context-dependent linguistic features encoded by the PLM. Based on the
alignment, the latent acoustic feature is forced to encode context dependent
linguistic information. We integrate this latent acoustic feature to build
conformer encoder-based CTC ASR system. On the AISHELL-1 data corpus, our
system achieved 3.96% and 4.27% character error rate (CER) for dev and test
sets, respectively, which corresponds to relative improvements of 28.39% and
29.42% compared to the baseline conformer CTC ASR system without cross-modal
knowledge transfer.
