  That shared features between train and test data are required for
generalisation in artificial neural networks has been a common assumption of
both proponents and critics of these models. Here, we show that convolutional
architectures avoid this limitation by applying them to two well known
challenges, based on learning the identity function and learning rules
governing sequences of words. In each case, successful performance on the test
set requires generalising to features that were not present in the training
data, which is typically not feasible for standard connectionist models.
However, our experiments demonstrate that neural networks can succeed on such
problems when they incorporate the weight sharing employed by convolutional
architectures. In the image processing domain, such architectures are intended
to reflect the symmetry under spatial translations of the natural world that
such images depict. We discuss the role of symmetry in the two tasks and its
connection to generalisation.
