  We present a joint camera and radar approach to enable autonomous vehicles to
understand and react to human gestures in everyday traffic. Initially, we
process the radar data with a PointNet followed by a spatio-temporal multilayer
perceptron (stMLP). Independently, the human body pose is extracted from the
camera frame and processed with a separate stMLP network. We propose a fusion
neural network for both modalities, including an auxiliary loss for each
modality. In our experiments with a collected dataset, we show the advantages
of gesture recognition with two modalities. Motivated by adverse weather
conditions, we also demonstrate promising performance when one of the sensors
lacks functionality.
