  We introduce three memory-augmented Recurrent Neural Networks (MARNNs) and
explore their capabilities on a series of simple language modeling tasks whose
solutions require stack-based mechanisms. We provide the first demonstration of
neural networks recognizing the generalized Dyck languages, which express the
core of what it means to be a language with hierarchical structure. Our
memory-augmented architectures are easy to train in an end-to-end fashion and
can learn the Dyck languages over as many as six parenthesis-pairs, in addition
to two deterministic palindrome languages and the string-reversal transduction
task, by emulating pushdown automata. Our experiments highlight the increased
modeling capacity of memory-augmented models over simple RNNs, while inflecting
our understanding of the limitations of these models.
