  Distributional reinforcement learning (RL) -- in which agents learn about all
the possible long-term consequences of their actions, and not just the expected
value -- is of great recent interest. One of the most important affordances of
a distributional view is facilitating a modern, measured, approach to risk when
outcomes are not completely certain. By contrast, psychological and
neuroscientific investigations into decision making under risk have utilized a
variety of more venerable theoretical models such as prospect theory that lack
axiomatically desirable properties such as coherence. Here, we consider a
particularly relevant risk measure for modeling human and animal planning,
called conditional value-at-risk (CVaR), which quantifies worst-case outcomes
(e.g., vehicle accidents or predation). We first adopt a conventional
distributional approach to CVaR in a sequential setting and reanalyze the
choices of human decision-makers in the well-known two-step task, revealing
substantial risk aversion that had been lurking under stickiness and
perseveration. We then consider a further critical property of risk
sensitivity, namely time consistency, showing alternatives to this form of CVaR
that enjoy this desirable characteristic. We use simulations to examine
settings in which the various forms differ in ways that have implications for
human and animal planning and behavior.
