  In model-based reinforcement learning (RL), an agent can leverage a learned
model to improve its way of behaving in different ways. Two of the prevalent
ways to do this are through decision-time and background planning methods. In
this study, we are interested in understanding how the value-based versions of
these two planning methods will compare against each other across different
settings. Towards this goal, we first consider the simplest instantiations of
value-based decision-time and background planning methods and provide
theoretical results on which one will perform better in the regular RL and
transfer learning settings. Then, we consider the modern instantiations of them
and provide hypotheses on which one will perform better in the same settings.
Finally, we perform illustrative experiments to validate these theoretical
results and hypotheses. Overall, our findings suggest that even though
value-based versions of the two planning methods perform on par in their
simplest instantiations, the modern instantiations of value-based decision-time
planning methods can perform on par or better than the modern instantiations of
value-based background planning methods in both the regular RL and transfer
learning settings.
