  Safety and performance are often two competing objectives in sequential
decision-making problems. Existing performant controllers, such as controllers
derived from reinforcement learning algorithms, often fall short of safety
guarantees. On the contrary, controllers that guarantee safety, such as those
derived from classical control theory, require restrictive assumptions and are
often conservative in performance. Our goal is to blend a performant and a safe
controller to generate a single controller that is safer than the performant
and accumulates higher rewards than the safe controller. To this end, we
propose a blending algorithm using the framework of contextual multi-armed
multi-objective bandits. At each stage, the algorithm observes the
environment's current context alongside an immediate reward and cost, which is
the underlying safety measure. The algorithm then decides which controller to
employ based on its observations. We demonstrate that the algorithm achieves
sublinear Pareto regret, a performance measure that models coherence with an
expert that always avoids picking the controller with both inferior safety and
performance. We derive an upper bound on the loss in individual objectives,
which imposes no additional computational complexity. We empirically
demonstrate the algorithm's success in blending a safe and a performant
controller in a safety-focused testbed, the Safety Gym environment. A
statistical analysis of the blended controller's total reward and cost reflects
two key takeaways: The blended controller shows a strict improvement in
performance compared to the safe controller, and it is safer than the
performant controller.
