  To let the state-of-the-art end-to-end ASR model enjoy data efficiency, as
well as much more unpaired text data by multi-modal training, one needs to
address two problems: 1) the synchronicity of feature sampling rates between
speech and language (aka text data); 2) the homogeneity of the learned
representations from two encoders. In this paper we propose to employ a novel
bidirectional attention mechanism (BiAM) to jointly learn both ASR encoder
(bottom layers) and text encoder with a multi-modal learning method. The BiAM
is to facilitate feature sampling rate exchange, realizing the quality of the
transformed features for the one kind to be measured in another space, with
diversified objective functions. As a result, the speech representations are
enriched with more linguistic information, while the representations generated
by the text encoder are more similar to corresponding speech ones, and
therefore the shared ASR models are more amenable for unpaired text data
pretraining. To validate the efficacy of the proposed method, we perform two
categories of experiments with or without extra unpaired text data.
Experimental results on Librispeech corpus show it can achieve up to 6.15% word
error rate reduction (WERR) with only paired data learning, while 9.23% WERR
when more unpaired text data is employed.
