  We study safe policy improvement (SPI) for partially observable Markov
decision processes (POMDPs). SPI is an offline reinforcement learning (RL)
problem that assumes access to (1) historical data about an environment, and
(2) the so-called behavior policy that previously generated this data by
interacting with the environment. SPI methods neither require access to a model
nor the environment itself, and aim to reliably improve the behavior policy in
an offline manner. Existing methods make the strong assumption that the
environment is fully observable. In our novel approach to the SPI problem for
POMDPs, we assume that a finite-state controller (FSC) represents the behavior
policy and that finite memory is sufficient to derive optimal policies. This
assumption allows us to map the POMDP to a finite-state fully observable MDP,
the history MDP. We estimate this MDP by combining the historical data and the
memory of the FSC, and compute an improved policy using an off-the-shelf SPI
algorithm. The underlying SPI method constrains the policy-space according to
the available data, such that the newly computed policy only differs from the
behavior policy when sufficient data was available. We show that this new
policy, converted into a new FSC for the (unknown) POMDP, outperforms the
behavior policy with high probability. Experimental results on several
well-established benchmarks show the applicability of the approach, even in
cases where finite memory is not sufficient.
