  For survival, a living agent must have the ability to assess risk (1) by
temporally anticipating accidents before they occur, and (2) by spatially
localizing risky regions in the environment to move away from threats. In this
paper, we take an agent-centric approach to study the accident anticipation and
risky region localization tasks. We propose a novel soft-attention Recurrent
Neural Network (RNN) which explicitly models both spatial and appearance-wise
non-linear interaction between the agent triggering the event and another agent
or static-region involved. In order to test our proposed method, we introduce
the Epic Fail (EF) dataset consisting of 3000 viral videos capturing various
accidents. In the experiments, we evaluate the risk assessment accuracy both in
the temporal domain (accident anticipation) and spatial domain (risky region
localization) on our EF dataset and the Street Accident (SA) dataset. Our
method consistently outperforms other baselines on both datasets.
