  Gaining a better understanding of how and what machine learning systems learn
is important to increase confidence in their decisions and catalyze further
research. In this paper, we analyze the predictions made by a specific type of
recurrent neural network, mixture density RNNs (MD-RNNs). These networks learn
to model predictions as a combination of multiple Gaussian distributions,
making them particularly interesting for problems where a sequence of inputs
may lead to several distinct future possibilities. An example is learning
internal models of an environment, where different events may or may not occur,
but where the average over different events is not meaningful. By analyzing the
predictions made by trained MD-RNNs, we find that their different Gaussian
components have two complementary roles: 1) Separately modeling different
stochastic events and 2) Separately modeling scenarios governed by different
rules. These findings increase our understanding of what is learned by
predictive MD-RNNs, and open up new research directions for further
understanding how we can benefit from their self-organizing model
decomposition.
