  Transfer in Reinforcement Learning (RL) refers to the idea of applying
knowledge gained from previous tasks to solving related tasks. Learning a
universal value function (Schaul et al., 2015), which generalizes over goals
and states, has previously been shown to be useful for transfer. However,
successor features are believed to be more suitable than values for transfer
(Dayan, 1993; Barreto et al.,2017), even though they cannot directly generalize
to new goals. In this paper, we propose (1) Universal Successor Features (USFs)
to capture the underlying dynamics of the environment while allowing
generalization to unseen goals and (2) a flexible end-to-end model of USFs that
can be trained by interacting with the environment. We show that learning USFs
is compatible with any RL algorithm that learns state values using a temporal
difference method. Our experiments in a simple gridworld and with two MuJoCo
environments show that USFs can greatly accelerate training when learning
multiple tasks and can effectively transfer knowledge to new tasks.
