  In this paper, we trace the history of neural networks applied to natural
language understanding tasks, and identify key contributions which the nature
of language has made to the development of neural network architectures. We
focus on the importance of variable binding and its instantiation in
attention-based models, and argue that Transformer is not a sequence model but
an induced-structure model. This perspective leads to predictions of the
challenges facing research in deep learning architectures for natural language
understanding.
