  Planning and reinforcement learning are two key approaches to sequential
decision making. Multi-step approximate real-time dynamic programming, a
recently successful algorithm class of which AlphaZero [Silver et al., 2018] is
an example, combines both by nesting planning within a learning loop. However,
the combination of planning and learning introduces a new question: how should
we balance time spend on planning, learning and acting? The importance of this
trade-off has not been explicitly studied before. We show that it is actually
of key importance, with computational results indicating that we should neither
plan too long nor too short. Conceptually, we identify a new spectrum of
planning-learning algorithms which ranges from exhaustive search (long
planning) to model-free RL (no planning), with optimal performance achieved
midway.
