  In this paper, a taxonomy for memory networks is proposed based on their
memory organization. The taxonomy includes all the popular memory networks:
vanilla recurrent neural network (RNN), long short term memory (LSTM ), neural
stack and neural Turing machine and their variants. The taxonomy puts all these
networks under a single umbrella and shows their relative expressive power ,
i.e. vanilla RNN <=LSTM<=neural stack<=neural RAM. The differences and
commonality between these networks are analyzed. These differences are also
connected to the requirements of different tasks which can give the user
instructions of how to choose or design an appropriate memory network for a
specific task. As a conceptual simplified class of problems, four tasks of
synthetic symbol sequences: counting, counting with interference, reversing and
repeat counting are developed and tested to verify our arguments. And we use
two natural language processing problems to discuss how this taxonomy helps
choosing the appropriate neural memory networks for real world problem.
