  Auxiliary tasks improve the representations learned by deep reinforcement
learning agents. Analytically, their effect is reasonably well understood; in
practice, however, their primary use remains in support of a main learning
objective, rather than as a method for learning representations. This is
perhaps surprising given that many auxiliary tasks are defined procedurally,
and hence can be treated as an essentially infinite source of information about
the environment. Based on this observation, we study the effectiveness of
auxiliary tasks for learning rich representations, focusing on the setting
where the number of tasks and the size of the agent's network are
simultaneously increased. For this purpose, we derive a new family of auxiliary
tasks based on the successor measure. These tasks are easy to implement and
have appealing theoretical properties. Combined with a suitable off-policy
learning rule, the result is a representation learning algorithm that can be
understood as extending Mahadevan & Maggioni (2007)'s proto-value functions to
deep reinforcement learning -- accordingly, we call the resulting object
proto-value networks. Through a series of experiments on the Arcade Learning
Environment, we demonstrate that proto-value networks produce rich features
that may be used to obtain performance comparable to established algorithms,
using only linear approximation and a small number (~4M) of interactions with
the environment's reward function.
