  We introduce Transformer Grammars (TGs), a novel class of Transformer
language models that combine (i) the expressive power, scalability, and strong
performance of Transformers and (ii) recursive syntactic compositions, which
here are implemented through a special attention mask and deterministic
transformation of the linearized tree. We find that TGs outperform various
strong baselines on sentence-level language modeling perplexity, as well as on
multiple syntax-sensitive language modeling evaluation metrics. Additionally,
we find that the recursive syntactic composition bottleneck which represents
each sentence as a single vector harms perplexity on document-level language
modeling, providing evidence that a different kind of memory mechanism -- one
that is independent of composed syntactic representations -- plays an important
role in current successful models of long text.
