  In certain situations, neural networks will represent environment states in
their hidden activations. Our goal is to visualize what environment states the
networks are representing. We experiment with a recurrent neural network (RNN)
architecture with a decoder network at the end. After training, we apply the
decoder to the intermediate representations of the network to visualize what
they represent. We define a quantitative interpretability metric and use it to
demonstrate that hidden states can be highly interpretable on a simple task. We
also develop autoencoder and adversarial techniques and show that benefit
interpretability.
