  Retrosynthesis, which aims to find a route to synthesize a target molecule
from commercially available starting materials, is a critical task in drug
discovery and materials design. Recently, the combination of ML-based
single-step reaction predictors with multi-step planners has led to promising
results. However, the single-step predictors are mostly trained offline to
optimize the single-step accuracy, without considering complete routes. Here,
we leverage reinforcement learning (RL) to improve the single-step predictor,
by using a tree-shaped MDP to optimize complete routes. Specifically, we
propose a novel online training algorithm, called Planning with Dual Value
Networks (PDVN), which alternates between the planning phase and updating
phase. In PDVN, we construct two separate value networks to predict the
synthesizability and cost of molecules, respectively. To maintain the
single-step accuracy, we design a two-branch network structure for the
single-step predictor. On the widely-used USPTO dataset, our PDVN algorithm
improves the search success rate of existing multi-step planners (e.g.,
increasing the success rate from 85.79% to 98.95% for Retro*, and reducing the
number of model calls by half while solving 99.47% molecules for RetroGraph).
Additionally, PDVN helps find shorter synthesis routes (e.g., reducing the
average route length from 5.76 to 4.83 for Retro*, and from 5.63 to 4.78 for
RetroGraph). Our code is available at \url{https://github.com/DiXue98/PDVN}.
