  We propose BlackOut, an approximation algorithm to efficiently train massive
recurrent neural network language models (RNNLMs) with million word
vocabularies. BlackOut is motivated by using a discriminative loss, and we
describe a new sampling strategy which significantly reduces computation while
improving stability, sample efficiency, and rate of convergence. One way to
understand BlackOut is to view it as an extension of the DropOut strategy to
the output layer, wherein we use a discriminative training loss and a weighted
sampling scheme. We also establish close connections between BlackOut,
importance sampling, and noise contrastive estimation (NCE). Our experiments,
on the recently released one billion word language modeling benchmark,
demonstrate scalability and accuracy of BlackOut; we outperform the
state-of-the art, and achieve the lowest perplexity scores on this dataset.
Moreover, unlike other established methods which typically require GPUs or CPU
clusters, we show that a carefully implemented version of BlackOut requires
only 1-10 days on a single machine to train a RNNLM with a million word
vocabulary and billions of parameters on one billion words. Although we
describe BlackOut in the context of RNNLM training, it can be used to any
networks with large softmax output layers.
