  Large-scale AI systems that combine search and learning have reached
super-human levels of performance in game-playing, but have also been shown to
fail in surprising ways. The brittleness of such models limits their efficacy
and trustworthiness in real-world deployments. In this work, we systematically
study one such algorithm, AlphaZero, and identify two phenomena related to the
nature of exploration. First, we find evidence of policy-value misalignment --
for many states, AlphaZero's policy and value predictions contradict each
other, revealing a tension between accurate move-selection and value estimation
in AlphaZero's objective. Further, we find inconsistency within AlphaZero's
value function, which causes it to generalize poorly, despite its policy
playing an optimal strategy. From these insights we derive VISA-VIS: a novel
method that improves policy-value alignment and value robustness in AlphaZero.
Experimentally, we show that our method reduces policy-value misalignment by up
to 76%, reduces value generalization error by up to 50%, and reduces average
value error by up to 55%.
