  LSTMs are powerful tools for modeling contextual information, as evidenced by
their success at the task of language modeling. However, modeling contexts in
very high dimensional space can lead to poor generalizability. We introduce the
Pyramidal Recurrent Unit (PRU), which enables learning representations in high
dimensional space with more generalization power and fewer parameters. PRUs
replace the linear transformation in LSTMs with more sophisticated interactions
including pyramidal and grouped linear transformations. This architecture gives
strong results on word-level language modeling while reducing the number of
parameters significantly. In particular, PRU improves the perplexity of a
recent state-of-the-art language model Merity et al. (2018) by up to 1.3 points
while learning 15-20% fewer parameters. For similar number of model parameters,
PRU outperforms all previous RNN models that exploit different gating
mechanisms and transformations. We provide a detailed examination of the PRU
and its behavior on the language modeling tasks. Our code is open-source and
available at https://sacmehta.github.io/PRU/
