  In spoken language understanding (SLU), what the user says is converted to
his/her intent. Recent work on end-to-end SLU has shown that accuracy can be
improved via pre-training approaches. We revisit ideas presented by Lugosch et
al. using speech pre-training and three-module modeling; however, to ease
construction of the end-to-end SLU model, we use as our phoneme module an
open-source acoustic-phonetic model from a DNN-HMM hybrid automatic speech
recognition (ASR) system instead of training one from scratch. Hence we
fine-tune on speech only for the word module, and we apply multi-target
learning (MTL) on the word and intent modules to jointly optimize SLU
performance. MTL yields a relative reduction of 40% in intent-classification
error rates (from 1.0% to 0.6%). Note that our three-module model is a
streaming method. The final outcome of the proposed three-module modeling
approach yields an intent accuracy of 99.4% on FluentSpeech, an intent error
rate reduction of 50% compared to that of Lugosch et al. Although we focus on
real-time streaming methods, we also list non-streaming methods for comparison.
