  Emergent-scene safety is the key milestone for fully autonomous driving, and
reliable on-time prediction is essential to maintain safety in emergency
scenarios. However, these emergency scenarios are long-tailed and hard to
collect, which restricts the system from getting reliable predictions. In this
paper, we build a new dataset, which aims at the long-term prediction with the
inconspicuous state variation in history for the emergency event, named the
Extro-Spective Prediction (ESP) problem. Based on the proposed dataset, a
flexible feature encoder for ESP is introduced to various prediction methods as
a seamless plug-in, and its consistent performance improvement underscores its
efficacy. Furthermore, a new metric named clamped temporal error (CTE) is
proposed to give a more comprehensive evaluation of prediction performance,
especially in time-sensitive emergency events of subseconds. Interestingly, as
our ESP features can be described in human-readable language naturally, the
application of integrating into ChatGPT also shows huge potential. The
ESP-dataset and all benchmarks are released at
https://dingrui-wang.github.io/ESP-Dataset/.
