  In this paper we investigate the properties of representations learned by
deep reinforcement learning systems. Much of the early work on representations
for reinforcement learning focused on designing fixed-basis architectures to
achieve properties thought to be desirable, such as orthogonality and sparsity.
In contrast, the idea behind deep reinforcement learning methods is that the
agent designer should not encode representational properties, but rather that
the data stream should determine the properties of the representation -- good
representations emerge under appropriate training schemes. In this paper we
bring these two perspectives together, empirically investigating the properties
of representations that support transfer in reinforcement learning. We
introduce and measure six representational properties over more than 25
thousand agent-task settings. We consider Deep Q-learning agents with different
auxiliary losses in a pixel-based navigation environment, with source and
transfer tasks corresponding to different goal locations. We develop a method
to better understand why some representations work better for transfer, through
a systematic approach varying task similarity and measuring and correlating
representation properties with transfer performance. We demonstrate the
generality of the methodology by investigating representations learned by a
Rainbow agent that successfully transfer across games modes in Atari 2600.
