  We propose a new perspective on representation learning in reinforcement
learning based on geometric properties of the space of value functions. We
leverage this perspective to provide formal evidence regarding the usefulness
of value functions as auxiliary tasks. Our formulation considers adapting the
representation to minimize the (linear) approximation of the value function of
all stationary policies for a given environment. We show that this optimization
reduces to making accurate predictions regarding a special class of value
functions which we call adversarial value functions (AVFs). We demonstrate that
using value functions as auxiliary tasks corresponds to an expected-error
relaxation of our formulation, with AVFs a natural candidate, and identify a
close relationship with proto-value functions (Mahadevan, 2005). We highlight
characteristics of AVFs and their usefulness as auxiliary tasks in a series of
experiments on the four-room domain.
