  Learning models of the environment from data is often viewed as an essential
component to building intelligent reinforcement learning (RL) agents. The
common practice is to separate the learning of the model from its use, by
constructing a model of the environment's dynamics that correctly predicts the
observed state transitions. In this paper we argue that the limited
representational resources of model-based RL agents are better used to build
models that are directly useful for value-based planning. As our main
contribution, we introduce the principle of value equivalence: two models are
value equivalent with respect to a set of functions and policies if they yield
the same Bellman updates. We propose a formulation of the model learning
problem based on the value equivalence principle and analyze how the set of
feasible solutions is impacted by the choice of policies and functions.
Specifically, we show that, as we augment the set of policies and functions
considered, the class of value equivalent models shrinks, until eventually
collapsing to a single point corresponding to a model that perfectly describes
the environment. In many problems, directly modelling state-to-state
transitions may be both difficult and unnecessary. By leveraging the
value-equivalence principle one may find simpler models without compromising
performance, saving computation and memory. We illustrate the benefits of
value-equivalent model learning with experiments comparing it against more
traditional counterparts like maximum likelihood estimation. More generally, we
argue that the principle of value equivalence underlies a number of recent
empirical successes in RL, such as Value Iteration Networks, the Predictron,
Value Prediction Networks, TreeQN, and MuZero, and provides a first theoretical
underpinning of those results.
