  We study the problem of Safe Policy Improvement (SPI) under constraints in
the offline Reinforcement Learning (RL) setting. We consider the scenario
where: (i) we have a dataset collected under a known baseline policy, (ii)
multiple reward signals are received from the environment inducing as many
objectives to optimize. We present an SPI formulation for this RL setting that
takes into account the preferences of the algorithm's user for handling the
trade-offs for different reward signals while ensuring that the new policy
performs at least as well as the baseline policy along each individual
objective. We build on traditional SPI algorithms and propose a novel method
based on Safe Policy Iteration with Baseline Bootstrapping (SPIBB, Laroche et
al., 2019) that provides high probability guarantees on the performance of the
agent in the true environment. We show the effectiveness of our method on a
synthetic grid-world safety task as well as in a real-world critical care
context to learn a policy for the administration of IV fluids and vasopressors
to treat sepsis.
