  Over the last few years, we have not seen any major developments in
model-free or model-based learning methods that would make one obsolete
relative to the other. In most cases, the used technique is heavily dependent
on the use case scenario or other attributes, e.g. the environment. Both
approaches have their own advantages, for example, sample efficiency or
computational efficiency. However, when combining the two, the advantages of
each can be combined and hence achieve better performance. The TD-MPC framework
is an example of this approach. On the one hand, a world model in combination
with model predictive control is used to get a good initial estimate of the
value function. On the other hand, a Q function is used to provide a good
long-term estimate. Similar to algorithms like MuZero a latent state
representation is used, where only task-relevant information is encoded to
reduce the complexity. In this paper, we propose the use of a reconstruction
function within the TD-MPC framework, so that the agent can reconstruct the
original observation given the internal state representation. This allows our
agent to have a more stable learning signal during training and also improves
sample efficiency. Our proposed addition of another loss term leads to improved
performance on both state- and image-based tasks from the DeepMind-Control
suite.
