  Forecasting pedestrians' future motions is essential for autonomous driving
systems to safely navigate in urban areas. However, existing prediction
algorithms often overly rely on past observed trajectories and tend to fail
around abrupt dynamic changes, such as when pedestrians suddenly start or stop
walking. We suggest that predicting these highly non-linear transitions should
form a core component to improve the robustness of motion prediction
algorithms. In this paper, we introduce the new task of pedestrian stop and go
forecasting. Considering the lack of suitable existing datasets for it, we
release TRANS, a benchmark for explicitly studying the stop and go behaviors of
pedestrians in urban traffic. We build it from several existing datasets
annotated with pedestrians' walking motions, in order to have various scenarios
and behaviors. We also propose a novel hybrid model that leverages
pedestrian-specific and scene features from several modalities, both video
sequences and high-level attributes, and gradually fuses them to integrate
multiple levels of context. We evaluate our model and several baselines on
TRANS, and set a new benchmark for the community to work on pedestrian stop and
go forecasting.
