  The Tactical Driver Behavior modeling problem requires understanding of
driver actions in complicated urban scenarios from a rich multi modal signals
including video, LiDAR and CAN bus data streams. However, the majority of deep
learning research is focused either on learning the vehicle/environment state
(sensor fusion) or the driver policy (from temporal data), but not both.
Learning both tasks end-to-end offers the richest distillation of knowledge,
but presents challenges in formulation and successful training. In this work,
we propose promising first steps in this direction. Inspired by the gating
mechanisms in LSTM, we propose gated recurrent fusion units (GRFU) that learn
fusion weighting and temporal weighting simultaneously. We demonstrate it's
superior performance over multimodal and temporal baselines in supervised
regression and classification tasks, all in the realm of autonomous navigation.
We note a 10% improvement in the mAP score over state-of-the-art for tactical
driver behavior classification in HDD dataset and a 20% drop in overall Mean
squared error for steering action regression on TORCS dataset.
