  Compared to hybrid automatic speech recognition (ASR) systems that use a
modular architecture in which each component can be independently adapted to a
new domain, recent end-to-end (E2E) ASR system are harder to customize due to
their all-neural monolithic construction. In this paper, we propose a novel
text representation and training framework for E2E ASR models. With this
approach, we show that a trained RNN Transducer (RNN-T) model's internal LM
component can be effectively adapted with text-only data. An RNN-T model
trained using both speech and text inputs improves over a baseline model
trained on just speech with close to 13% word error rate (WER) reduction on the
Switchboard and CallHome test sets of the NIST Hub5 2000 evaluation. The
usefulness of the proposed approach is further demonstrated by customizing this
general purpose RNN-T model to three separate datasets. We observe 20-45%
relative word error rate (WER) reduction in these settings with this novel LM
style customization technique using only unpaired text data from the new
domains.
