  In the last decade, deep artificial neural networks have achieved astounding
performance in many natural language processing tasks. Given the high
productivity of language, these models must possess effective generalization
abilities. It is widely assumed that humans handle linguistic productivity by
means of algebraic compositional rules: Are deep networks similarly
compositional? After reviewing the main innovations characterizing current deep
language processing networks, I discuss a set of studies suggesting that deep
networks are capable of subtle grammar-dependent generalizations, but also that
they do not rely on systematic compositional rules. I argue that the intriguing
behaviour of these devices (still awaiting a full understanding) should be of
interest to linguists and cognitive scientists, as it offers a new perspective
on possible computational strategies to deal with linguistic productivity
beyond rule-based compositionality, and it might lead to new insights into the
less systematic generalization patterns that also appear in natural language.
