  In dynamic programming and reinforcement learning, the policy for the
sequential decision making of an agent in a stochastic environment is usually
determined by expressing the goal as a scalar reward function and seeking a
policy that maximizes the expected total reward. However, many goals that
humans care about naturally concern multiple aspects of the world, and it may
not be obvious how to condense those into a single reward function.
Furthermore, maximization suffers from specification gaming, where the obtained
policy achieves a high expected total reward in an unintended way, often taking
extreme or nonsensical actions.
  Here we consider finite acyclic Markov Decision Processes with multiple
distinct evaluation metrics, which do not necessarily represent quantities that
the user wants to be maximized. We assume the task of the agent is to ensure
that the vector of expected totals of the evaluation metrics falls into some
given convex set, called the aspiration set. Our algorithm guarantees that this
task is fulfilled by using simplices to approximate feasibility sets and
propagate aspirations forward while ensuring they remain feasible. It has
complexity linear in the number of possible state-action-successor triples and
polynomial in the number of evaluation metrics. Moreover, the explicitly
non-maximizing nature of the chosen policy and goals yields additional degrees
of freedom, which can be used to apply heuristic safety criteria to the choice
of actions. We discuss several such safety criteria that aim to steer the agent
towards more conservative behavior.
