  We consider the problem of learning models for risk-sensitive reinforcement
learning. We theoretically demonstrate that proper value equivalence, a method
of learning models which can be used to plan optimally in the risk-neutral
setting, is not sufficient to plan optimally in the risk-sensitive setting. We
leverage distributional reinforcement learning to introduce two new notions of
model equivalence, one which is general and can be used to plan for any risk
measure, but is intractable; and a practical variation which allows one to
choose which risk measures they may plan optimally for. We demonstrate how our
framework can be used to augment any model-free risk-sensitive algorithm, and
provide both tabular and large-scale experiments to demonstrate its ability.
