  Policy optimization is among the most popular and successful reinforcement
learning algorithms, and there is increasing interest in understanding its
theoretical guarantees. In this work, we initiate the study of policy
optimization for the stochastic shortest path (SSP) problem, a goal-oriented
reinforcement learning model that strictly generalizes the finite-horizon model
and better captures many applications. We consider a wide range of settings,
including stochastic and adversarial environments under full information or
bandit feedback, and propose a policy optimization algorithm for each setting
that makes use of novel correction terms and/or variants of dilated bonuses
(Luo et al., 2021). For most settings, our algorithm is shown to achieve a
near-optimal regret bound.
  One key technical contribution of this work is a new approximation scheme to
tackle SSP problems that we call \textit{stacked discounted approximation} and
use in all our proposed algorithms. Unlike the finite-horizon approximation
that is heavily used in recent SSP algorithms, our new approximation enables us
to learn a near-stationary policy with only logarithmic changes during an
episode and could lead to an exponential improvement in space complexity.
