  This paper proposed a class of novel Deep Recurrent Neural Networks which can
incorporate language-level information into acoustic models. For simplicity, we
named these networks Recurrent Deep Language Networks (RDLNs). Multiple
variants of RDLNs were considered, including two kinds of context information,
two methods to process the context, and two methods to incorporate the
language-level information. RDLNs provided possible methods to fine-tune the
whole Automatic Speech Recognition (ASR) system in the acoustic modeling
process.
