  Predicting pedestrian behavior is a crucial task for intelligent driving
systems. Accurate predictions require a deep understanding of various
contextual elements that potentially impact the way pedestrians behave. To
address this challenge, we propose a novel framework that relies on different
data modalities to predict future trajectories and crossing actions of
pedestrians from an ego-centric perspective. Specifically, our model utilizes a
cross-modal Transformer architecture to capture dependencies between different
data types. The output of the Transformer is augmented with representations of
interactions between pedestrians and other traffic agents conditioned on the
pedestrian and ego-vehicle dynamics that are generated via a semantic attentive
interaction module. Lastly, the context encodings are fed into a multi-stream
decoder framework using a gated-shared network. We evaluate our algorithm on
public pedestrian behavior benchmarks, PIE and JAAD, and show that our model
improves state-of-the-art in trajectory and action prediction by up to 22% and
13% respectively on various metrics. The advantages brought by components of
our model are investigated via extensive ablation studies.
