  Real-world planning problems, including autonomous driving and sustainable
energy applications like carbon storage and resource exploration, have recently
been modeled as partially observable Markov decision processes (POMDPs) and
solved using approximate methods. To solve high-dimensional POMDPs in practice,
state-of-the-art methods use online planning with problem-specific heuristics
to reduce planning horizons and make the problems tractable. Algorithms that
learn approximations to replace heuristics have recently found success in
large-scale fully observable domains. The key insight is the combination of
online Monte Carlo tree search with offline neural network approximations of
the optimal policy and value function. In this work, we bring this insight to
partially observable domains and propose BetaZero, a belief-state planning
algorithm for high-dimensional POMDPs. BetaZero learns offline approximations
that replace heuristics to enable online decision making in long-horizon
problems. We address several challenges inherent in large-scale partially
observable domains; namely challenges of transitioning in stochastic
environments, prioritizing action branching with a limited search budget, and
representing beliefs as input to the network. To formalize the use of all
limited search information, we train against a novel $Q$-weighted visit counts
policy. We test BetaZero on various well-established POMDP benchmarks found in
the literature and a real-world problem of critical mineral exploration.
Experiments show that BetaZero outperforms state-of-the-art POMDP solvers on a
variety of tasks.
