  The impressive progress in NLP techniques has been driven by the development
of multi-task benchmarks such as GLUE and SuperGLUE. While these benchmarks
focus on tasks for one or two input sentences, there has been exciting work in
designing efficient techniques for processing much longer inputs. In this
paper, we present MuLD: a new long document benchmark consisting of only
documents over 10,000 tokens. By modifying existing NLP tasks, we create a
diverse benchmark which requires models to successfully model long-term
dependencies in the text. We evaluate how existing models perform, and find
that our benchmark is much more challenging than their `short document'
equivalents. Furthermore, by evaluating both regular and efficient
transformers, we show that models with increased context length are better able
to solve the tasks presented, suggesting that future improvements in these
models are vital for solving similar long document problems. We release the
data and code for baselines to encourage further research on efficient NLP
models.
