  Autonomous agents are limited in their ability to observe the world state.
Partially observable Markov decision processes (POMDPs) formally model the
problem of planning under world state uncertainty, but POMDPs with continuous
actions and nonlinear dynamics suitable for robotics applications are
challenging to solve. In this paper, we present an efficient differential
dynamic programming (DDP) algorithm for belief space planning in POMDPs with
uncertainty over a discrete latent state, and continuous states, actions,
observations, and nonlinear dynamics. This representation allows planning of
dynamic trajectories which are sensitive to structured uncertainty over
discrete latent world states. We develop dynamic programming techniques to
optimize a contingency plan over a tree of possible observations and belief
space trajectories, and also derive a hierarchical version of the algorithm.
Our method is applicable to problems with uncertainty over the cost or reward
function (e.g., the configuration of goals or obstacles), uncertainty over the
dynamics (e.g., the dynamical mode of a hybrid system), and uncertainty about
interactions, where other agents' behavior is conditioned on latent intentions.
Benchmarks show that our algorithm outperforms popular heuristic approaches to
planning under uncertainty, and results from an autonomous lane changing task
demonstrate that our algorithm can synthesize robust interactive trajectories.
