  In this paper we proposed a novel Adversarial Training (AT) approach for
end-to-end speech recognition using a Criticizing Language Model (CLM). In this
way the CLM and the automatic speech recognition (ASR) model can challenge and
learn from each other iteratively to improve the performance. Since the CLM
only takes the text as input, huge quantities of unpaired text data can be
utilized in this approach within end-to-end training. Moreover, AT can be
applied to any end-to-end ASR model using any deep-learning-based language
modeling frameworks, and compatible with any existing end-to-end decoding
method. Initial results with an example experimental setup demonstrated the
proposed approach is able to gain consistent improvements efficiently from
auxiliary text data under different scenarios.
