  Current language models are considered to have sub-human capabilities at
natural language tasks like question-answering or writing code. However,
language models are not trained to perform well at these tasks, they are
trained to accurately predict the next token given previous tokes in tokenized
text. It is not clear whether language models are better or worse than humans
at next token prediction. To try to answer this question, we performed two
distinct experiments to directly compare humans and language models on this
front: one measuring top-1 accuracy and the other measuring perplexity. In both
experiments, we find humans to be consistently \emph{worse} than even
relatively small language models like GPT3-Ada at next-token prediction.
