  In this paper, we propose a generative knowledge transfer technique that
trains an RNN based language model (student network) using text and output
probabilities generated from a previously trained RNN (teacher network). The
text generation can be conducted by either the teacher or the student network.
We can also improve the performance by taking the ensemble of soft labels
obtained from multiple teacher networks. This method can be used for privacy
conscious language model adaptation because no user data is directly used for
training. Especially, when the soft labels of multiple devices are aggregated
via a trusted third party, we can expect very strong privacy protection.
