  We introduce the forward-backward (FB) representation of the dynamics of a
reward-free Markov decision process. It provides explicit near-optimal policies
for any reward specified a posteriori. During an unsupervised phase, we use
reward-free interactions with the environment to learn two representations via
off-the-shelf deep learning methods and temporal difference (TD) learning. In
the test phase, a reward representation is estimated either from observations
or an explicit reward description (e.g., a target state). The optimal policy
for that reward is directly obtained from these representations, with no
planning. We assume access to an exploration scheme or replay buffer for the
first phase.
  The corresponding unsupervised loss is well-principled: if training is
perfect, the policies obtained are provably optimal for any reward function.
With imperfect training, the sub-optimality is proportional to the unsupervised
approximation error. The FB representation learns long-range relationships
between states and actions, via a predictive occupancy map, without having to
synthesize states as in model-based approaches.
  This is a step towards learning controllable agents in arbitrary black-box
stochastic environments. This approach compares well to goal-oriented RL
algorithms on discrete and continuous mazes, pixel-based MsPacman, and the
FetchReach virtual robot arm. We also illustrate how the agent can immediately
adapt to new tasks beyond goal-oriented RL.
