  In this paper, we explore an approach to auxiliary task discovery in
reinforcement learning based on ideas from representation learning. Auxiliary
tasks tend to improve data efficiency by forcing the agent to learn auxiliary
prediction and control objectives in addition to the main task of maximizing
reward, and thus producing better representations. Typically these tasks are
designed by people. Meta-learning offers a promising avenue for automatic task
discovery; however, these methods are computationally expensive and challenging
to tune in practice. In this paper, we explore a complementary approach to the
auxiliary task discovery: continually generating new auxiliary tasks and
preserving only those with high utility. We also introduce a new measure of
auxiliary tasks' usefulness based on how useful the features induced by them
are for the main task. Our discovery algorithm significantly outperforms random
tasks and learning without auxiliary tasks across a suite of environments.
