  Self-supervised learned (SSL) models such as Wav2vec and HuBERT yield
state-of-the-art results on speech-related tasks. Given the effectiveness of
such models, it is advantageous to use them in conventional ASR systems. While
some approaches suggest incorporating these models as a trainable encoder or a
learnable frontend, training such systems is extremely slow and requires a lot
of computation cycles. In this work, we propose two simple approaches that use
(1) framewise addition and (2) cross-attention mechanisms to efficiently
incorporate the representations from the SSL model(s) into the ASR
architecture, resulting in models that are comparable in size with standard
encoder-decoder conformer systems while also avoiding the usage of SSL models
during training. Our approach results in faster training and yields significant
performance gains on the Librispeech and Tedlium datasets compared to
baselines. We further provide detailed analysis and ablation studies that
demonstrate the effectiveness of our approach.
