  Recent advances in deep reinforcement learning have demonstrated the
capability of learning complex control policies from many types of
environments. When learning policies for safety-critical applications, it is
essential to be sensitive to risks and avoid catastrophic events. Towards this
goal, we propose an actor-critic framework that models the uncertainty of the
future and simultaneously learns a policy based on that uncertainty model.
Specifically, given a distribution of the future return for any state and
action, we optimize policies for varying levels of conditional Value-at-Risk.
The learned policy can map the same state to different actions depending on the
propensity for risk. We demonstrate the effectiveness of our approach in the
domain of driving simulations, where we learn maneuvers in two scenarios. Our
learned controller can dynamically select actions along a continuous axis,
where safe and conservative behaviors are found at one end while riskier
behaviors are found at the other. Finally, when testing with very different
simulation parameters, our risk-averse policies generalize significantly better
compared to other reinforcement learning approaches.
