  The underlying structure of natural language is hierarchical; words combine
into phrases, which in turn form clauses. An awareness of this hierarchical
structure can aid machine learning models in performing many linguistic tasks.
However, most such models just process text sequentially and there is no bias
towards learning hierarchical structure encoded into their architecture. In
this paper, we extend the recent transformer model (Vaswani et al., 2017) by
enabling it to learn hierarchical representations. To achieve this, we adapt
the ordering mechanism introduced in Shen et al., 2018, to the self-attention
module of the transformer architecture. We train our new model on language
modelling and then apply it to the task of unsupervised parsing. We achieve
reasonable results on the freely available subset of the WSJ10 dataset with an
F1-score of about 50%.
