  Designing reward functions for efficiently guiding reinforcement learning
(RL) agents toward specific behaviors is a complex task. This is challenging
since it requires the identification of reward structures that are not sparse
and that avoid inadvertently inducing undesirable behaviors. Naively modifying
the reward structure to offer denser and more frequent feedback can lead to
unintended outcomes and promote behaviors that are not aligned with the
designer's intended goal. Although potential-based reward shaping is often
suggested as a remedy, we systematically investigate settings where deploying
it often significantly impairs performance. To address these issues, we
introduce a new framework that uses a bi-level objective to learn
\emph{behavior alignment reward functions}. These functions integrate auxiliary
rewards reflecting a designer's heuristics and domain knowledge with the
environment's primary rewards. Our approach automatically determines the most
effective way to blend these types of feedback, thereby enhancing robustness
against heuristic reward misspecification. Remarkably, it can also adapt an
agent's policy optimization process to mitigate suboptimalities resulting from
limitations and biases inherent in the underlying RL algorithms. We evaluate
our method's efficacy on a diverse set of tasks, from small-scale experiments
to high-dimensional control challenges. We investigate heuristic auxiliary
rewards of varying quality -- some of which are beneficial and others
detrimental to the learning process. Our results show that our framework offers
a robust and principled way to integrate designer-specified heuristics. It not
only addresses key shortcomings of existing approaches but also consistently
leads to high-performing solutions, even when given misaligned or
poorly-specified auxiliary reward functions.
