  In Model-based Reinforcement Learning (MBRL), model learning is critical
since an inaccurate model can bias policy learning via generating misleading
samples. However, learning an accurate model can be difficult since the policy
is continually updated and the induced distribution over visited states used
for model learning shifts accordingly. Prior methods alleviate this issue by
quantifying the uncertainty of model-generated samples. However, these methods
only quantify the uncertainty passively after the samples were generated,
rather than foreseeing the uncertainty before model trajectories fall into
those highly uncertain regions. The resulting low-quality samples can induce
unstable learning targets and hinder the optimization of the policy. Moreover,
while being learned to minimize one-step prediction errors, the model is
generally used to predict for multiple steps, leading to a mismatch between the
objectives of model learning and model usage. To this end, we propose
\emph{Plan To Predict} (P2P), an MBRL framework that treats the model rollout
process as a sequential decision making problem by reversely considering the
model as a decision maker and the current policy as the dynamics. In this way,
the model can quickly adapt to the current policy and foresee the multi-step
future uncertainty when generating trajectories. Theoretically, we show that
the performance of P2P can be guaranteed by approximately optimizing a lower
bound of the true environment return. Empirical results demonstrate that P2P
achieves state-of-the-art performance on several challenging benchmark tasks.
