  Online planning is crucial for high performance in many complex sequential
decision-making tasks. Monte Carlo Tree Search (MCTS) employs a principled
mechanism for trading off exploration for exploitation for efficient online
planning, and it outperforms comparison methods in many discrete
decision-making domains such as Go, Chess, and Shogi. Subsequently, extensions
of MCTS to continuous domains have been developed. However, the inherent high
branching factor and the resulting explosion of the search tree size are
limiting the existing methods. To address this problem, we propose Continuous
Monte Carlo Graph Search (CMCGS), an extension of MCTS to online planning in
environments with continuous state and action spaces. CMCGS takes advantage of
the insight that, during planning, sharing the same action policy between
several states can yield high performance. To implement this idea, at each time
step, CMCGS clusters similar states into a limited number of stochastic action
bandit nodes, which produce a layered directed graph instead of an MCTS search
tree. Experimental evaluation shows that CMCGS outperforms comparable planning
methods in several complex continuous DeepMind Control Suite benchmarks and 2D
navigation and exploration tasks with limited sample budgets. Furthermore,
CMCGS can be scaled up through parallelization, and it outperforms the
Cross-Entropy Method (CEM) in continuous control with learned dynamics models.
