  One of the major challenges for autonomous vehicles in urban environments is
to understand and predict other road users' actions, in particular, pedestrians
at the point of crossing. The common approach to solving this problem is to use
the motion history of the agents to predict their future trajectories. However,
pedestrians exhibit highly variable actions most of which cannot be understood
without visual observation of the pedestrians themselves and their
surroundings. To this end, we propose a solution for the problem of pedestrian
action anticipation at the point of crossing. Our approach uses a novel stacked
RNN architecture in which information collected from various sources, both
scene dynamics and visual features, is gradually fused into the network at
different levels of processing. We show, via extensive empirical evaluations,
that the proposed algorithm achieves a higher prediction accuracy compared to
alternative recurrent network architectures. We conduct experiments to
investigate the impact of the length of observation, time to event and types of
features on the performance of the proposed method. Finally, we demonstrate how
different data fusion strategies impact prediction accuracy.
