  Zero-sum Markov Stackelberg games can be used to model myriad problems, in
domains ranging from economics to human robot interaction. In this paper, we
develop policy gradient methods that solve these games in continuous state and
action settings using noisy gradient estimates computed from observed
trajectories of play. When the games are convex-concave, we prove that our
algorithms converge to Stackelberg equilibrium in polynomial time. We also show
that reach-avoid problems are naturally modeled as convex-concave zero-sum
Markov Stackelberg games, and that Stackelberg equilibrium policies are more
effective than their Nash counterparts in these problems.
