  Deep reinforcement learning, applied to vision-based problems like Atari
games, maps pixels directly to actions; internally, the deep neural network
bears the responsibility of both extracting useful information and making
decisions based on it. By separating the image processing from decision-making,
one could better understand the complexity of each task, as well as potentially
find smaller policy representations that are easier for humans to understand
and may generalize better. To this end, we propose a new method for learning
policies and compact state representations separately but simultaneously for
policy approximation in reinforcement learning. State representations are
generated by an encoder based on two novel algorithms: Increasing Dictionary
Vector Quantization makes the encoder capable of growing its dictionary size
over time, to address new observations as they appear in an open-ended
online-learning context; Direct Residuals Sparse Coding encodes observations by
disregarding reconstruction error minimization, and aiming instead for highest
information inclusion. The encoder autonomously selects observations online to
train on, in order to maximize code sparsity. As the dictionary size increases,
the encoder produces increasingly larger inputs for the neural network: this is
addressed by a variation of the Exponential Natural Evolution Strategies
algorithm which adapts its probability distribution dimensionality along the
run. We test our system on a selection of Atari games using tiny neural
networks of only 6 to 18 neurons (depending on the game's controls). These are
still capable of achieving results comparable---and occasionally superior---to
state-of-the-art techniques which use two orders of magnitude more neurons.
