  In recent years, Recurrent Neural Networks (RNNs) based models have been
applied to the Slot Filling problem of Spoken Language Understanding and
achieved the state-of-the-art performances. In this paper, we investigate the
effect of incorporating pre-trained language models into RNN based Slot Filling
models. Our evaluation on the Airline Travel Information System (ATIS) data
corpus shows that we can significantly reduce the size of labeled training data
and achieve the same level of Slot Filling performance by incorporating extra
word embedding and language model embedding layers pre-trained on unlabeled
corpora.
