  Neural network-based language models deal with data sparsity problems by
mapping the large discrete space of words into a smaller continuous space of
real-valued vectors. By learning distributed vector representations for words,
each training sample informs the neural network model about a combinatorial
number of other patterns. In this paper, we exploit the sparsity in natural
language even further by encoding each unique input word using a fixed sparse
random representation. These sparse codes are then projected onto a smaller
embedding space which allows for the encoding of word occurrences from a
possibly unknown vocabulary, along with the creation of more compact language
models using a reduced number of parameters. We investigate the properties of
our encoding mechanism empirically, by evaluating its performance on the widely
used Penn Treebank corpus. We show that guaranteeing approximately equidistant
(nearly orthogonal) vector representations for unique discrete inputs is enough
to provide the neural network model with enough information to learn --and make
use-- of distributed representations for these inputs.
