  We explore the architecture of recurrent neural networks (RNNs) by studying
the complexity of string sequences it is able to memorize. Symbolic sequences
of different complexity are generated to simulate RNN training and study
parameter configurations with a view to the network's capability of learning
and inference. We compare Long Short-Term Memory (LSTM) networks and gated
recurrent units (GRUs). We find that an increase in RNN depth does not
necessarily result in better memorization capability when the training time is
constrained. Our results also indicate that the learning rate and the number of
units per layer are among the most important hyper-parameters to be tuned.
Generally, GRUs outperform LSTM networks on low-complexity sequences while on
high-complexity sequences LSTMs perform better.
