  A good state representation is crucial to solving complicated reinforcement
learning (RL) challenges. Many recent works focus on designing auxiliary losses
for learning informative representations. Unfortunately, these handcrafted
objectives rely heavily on expert knowledge and may be sub-optimal. In this
paper, we propose a principled and universal method for learning better
representations with auxiliary loss functions, named Automated Auxiliary Loss
Search (A2LS), which automatically searches for top-performing auxiliary loss
functions for RL. Specifically, based on the collected trajectory data, we
define a general auxiliary loss space of size $7.5 \times 10^{20}$ and explore
the space with an efficient evolutionary search strategy. Empirical results
show that the discovered auxiliary loss (namely, A2-winner) significantly
improves the performance on both high-dimensional (image) and low-dimensional
(vector) unseen tasks with much higher efficiency, showing promising
generalization ability to different settings and even different benchmark
domains. We conduct a statistical analysis to reveal the relations between
patterns of auxiliary losses and RL performance.
