  Projecting high-dimensional environment observations into lower-dimensional
structured representations can considerably improve data-efficiency for
reinforcement learning in domains with limited data such as robotics. Can a
single generally useful representation be found? In order to answer this
question, it is important to understand how the representation will be used by
the agent and what properties such a 'good' representation should have. In this
paper we systematically evaluate a number of common learnt and hand-engineered
representations in the context of three robotics tasks: lifting, stacking and
pushing of 3D blocks. The representations are evaluated in two use-cases: as
input to the agent, or as a source of auxiliary tasks. Furthermore, the value
of each representation is evaluated in terms of three properties:
dimensionality, observability and disentanglement. We can significantly improve
performance in both use-cases and demonstrate that some representations can
perform commensurate to simulator states as agent inputs. Finally, our results
challenge common intuitions by demonstrating that: 1) dimensionality strongly
matters for task generation, but is negligible for inputs, 2) observability of
task-relevant aspects mostly affects the input representation use-case, and 3)
disentanglement leads to better auxiliary tasks, but has only limited benefits
for input representations. This work serves as a step towards a more systematic
understanding of what makes a 'good' representation for control in robotics,
enabling practitioners to make more informed choices for developing new learned
or hand-engineered representations.
