  Recent studies have shown that a hybrid of self-attention networks (SANs) and
recurrent neural networks (RNNs) outperforms both individual architectures,
while not much is known about why the hybrid models work. With the belief that
modeling hierarchical structure is an essential complementary between SANs and
RNNs, we propose to further enhance the strength of hybrid models with an
advanced variant of RNNs - Ordered Neurons LSTM (ON-LSTM), which introduces a
syntax-oriented inductive bias to perform tree-like composition. Experimental
results on the benchmark machine translation task show that the proposed
approach outperforms both individual architectures and a standard hybrid model.
Further analyses on targeted linguistic evaluation and logical inference tasks
demonstrate that the proposed approach indeed benefits from a better modeling
of hierarchical structure.
