  It is well known that canonical recurrent neural networks (RNNs) face
limitations in learning long-term dependencies which have been addressed by
memory structures in long short-term memory (LSTM) networks. Neural Turing
machines (NTMs) are novel RNNs that implement the notion of programmable
computers with neural network controllers that can learn simple algorithmic
tasks. Matrix neural networks feature matrix representation which inherently
preserves the spatial structure of data when compared to canonical neural
networks that use vector-based representation. One may then argue that neural
networks with matrix representations may have the potential to provide better
memory capacity. In this paper, we define and study a probabilistic notion of
memory capacity based on Fisher information for matrix-based RNNs. We find
bounds on memory capacity for such networks under various hypotheses and
compare them with their vector counterparts. In particular, we show that the
memory capacity of such networks is bounded by $N^2$ for $N\times N$ state
matrix which generalizes the one known for vector networks. We also show and
analyze the increase in memory capacity for such networks which is introduced
when one exhibits an external state memory, such as NTMs. Consequently, we
construct NTMs with RNN controllers with matrix-based representation of
external memory, leading us to introduce Matrix NTMs. We demonstrate the
performance of this class of memory networks under certain algorithmic learning
tasks such as copying and recall and compare it with Matrix RNNs. We find an
improvement in the performance of Matrix NTMs by the addition of external
memory, in comparison to Matrix RNNs.
