  Existing multi-objective reinforcement learning (MORL) algorithms do not
account for objectives that arise from players with differing beliefs.
Concretely, consider two players with different beliefs and utility functions
who may cooperate to build a machine that takes actions on their behalf. A
representation is needed for how much the machine's policy will prioritize each
player's interests over time. Assuming the players have reached common
knowledge of their situation, this paper derives a recursion that any Pareto
optimal policy must satisfy. Two qualitative observations can be made from the
recursion: the machine must (1) use each player's own beliefs in evaluating how
well an action will serve that player's utility function, and (2) shift the
relative priority it assigns to each player's expected utilities over time, by
a factor proportional to how well that player's beliefs predict the machine's
inputs. Observation (2) represents a substantial divergence from na\"{i}ve
linear utility aggregation (as in Harsanyi's utilitarian theorem, and existing
MORL algorithms), which is shown here to be inadequate for Pareto optimal
sequential decision-making on behalf of players with different beliefs.
