  We present PaLM, a hybrid parser and neural language model. Building on an
RNN language model, PaLM adds an attention layer over text spans in the left
context. An unsupervised constituency parser can be derived from its attention
weights, using a greedy decoding algorithm. We evaluate PaLM on language
modeling, and empirically show that it outperforms strong baselines. If
syntactic annotations are available, the attention component can be trained in
a supervised manner, providing syntactically-informed representations of the
context, and further improving language modeling performance.
