  Many reinforcement-learning researchers treat the reward function as a part
of the environment, meaning that the agent can only know the reward of a state
if it encounters that state in a trial run. However, we argue that this is an
unnecessary limitation and instead, the reward function should be provided to
the learning algorithm. The advantage is that the algorithm can then use the
reward function to check the reward for states that the agent hasn't even
encountered yet. In addition, the algorithm can simultaneously learn policies
for multiple reward functions. For each state, the algorithm would calculate
the reward using each of the reward functions and add the rewards to its
experience replay dataset. The Hindsight Experience Replay algorithm developed
by Andrychowicz et al. (2017) does just this, and learns to generalize across a
distribution of sparse, goal-based rewards. We extend this algorithm to
linearly-weighted, multi-objective rewards and learn a single policy that can
generalize across all linear combinations of the multi-objective reward.
Whereas other multi-objective algorithms teach the Q-function to generalize
across the reward weights, our algorithm enables the policy to generalize, and
can thus be used with continuous actions.
