  This paper proposes a reinforcing method that refines the output layers of
existing Recurrent Neural Network (RNN) language models. We refer to our
proposed method as Input-to-Output Gate (IOG). IOG has an extremely simple
structure, and thus, can be easily combined with any RNN language models. Our
experiments on the Penn Treebank and WikiText-2 datasets demonstrate that IOG
consistently boosts the performance of several different types of current
topline RNN language models.
