  Just because some purely recurrent models suffer from being hard to optimize
and inefficient on today's hardware, they are not necessarily bad models of
language. We demonstrate this by the extent to which these models can still be
improved by a combination of a slightly better recurrent cell, architecture,
objective, as well as optimization. In the process, we establish a new state of
the art for language modelling on small datasets and on Enwik8 with dynamic
evaluation.
