  We propose Diverse Embedding Neural Network (DENN), a novel architecture for
language models (LMs). A DENNLM projects the input word history vector onto
multiple diverse low-dimensional sub-spaces instead of a single
higher-dimensional sub-space as in conventional feed-forward neural network
LMs. We encourage these sub-spaces to be diverse during network training
through an augmented loss function. Our language modeling experiments on the
Penn Treebank data set show the performance benefit of using a DENNLM.
