  We present a set of experiments to demonstrate that deep recurrent neural
networks (RNNs) learn internal representations that capture soft hierarchical
notions of syntax from highly varied supervision. We consider four syntax tasks
at different depths of the parse tree; for each word, we predict its part of
speech as well as the first (parent), second (grandparent) and third level
(great-grandparent) constituent labels that appear above it. These predictions
are made from representations produced at different depths in networks that are
pretrained with one of four objectives: dependency parsing, semantic role
labeling, machine translation, or language modeling. In every case, we find a
correspondence between network depth and syntactic depth, suggesting that a
soft syntactic hierarchy emerges. This effect is robust across all conditions,
indicating that the models encode significant amounts of syntax even in the
absence of an explicit syntactic training supervision.
