  With language modeling becoming the popular base task for unsupervised
representation learning in Natural Language Processing, it is important to come
up with new architectures and techniques for faster and better training of
language models. However, due to a peculiarity of languages -- the larger the
dataset, the higher the average number of times a word appears in that dataset
-- datasets of different sizes have very different properties. Architectures
performing well on small datasets might not perform well on larger ones. For
example, LSTM models perform well on WikiText-2 but poorly on WikiText-103,
while Transformer models perform well on WikiText-103 but not on WikiText-2.
For setups like architectural search, this is a challenge since it is
prohibitively costly to run a search on the full dataset but it is not
indicative to experiment on smaller ones. In this paper, we introduce
SimpleBooks, a small dataset with the average word frequency as high as that of
much larger ones. Created from 1,573 Gutenberg books with the highest ratio of
word-level book length to vocabulary size, SimpleBooks contains 92M word-level
tokens, on par with WikiText-103 (103M tokens), but has the vocabulary of 98K,
a third of WikiText-103's. SimpleBooks can be downloaded from
https://dldata-public.s3.us-east-2.amazonaws.com/simplebooks.zip.
