  This paper introduces a novel human pose estimation benchmark, Human Pose
with Millimeter Wave Radar (HuPR), that includes synchronized vision and radio
signal components. This dataset is created using cross-calibrated mmWave radar
sensors and a monocular RGB camera for cross-modality training of radar-based
human pose estimation. There are two advantages of using mmWave radar to
perform human pose estimation. First, it is robust to dark and low-light
conditions. Second, it is not visually perceivable by humans and thus, can be
widely applied to applications with privacy concerns, e.g., surveillance
systems in patient rooms. In addition to the benchmark, we propose a
cross-modality training framework that leverages the ground-truth 2D keypoints
representing human body joints for training, which are systematically generated
from the pre-trained 2D pose estimation network based on a monocular camera
input image, avoiding laborious manual label annotation efforts. The framework
consists of a new radar pre-processing method that better extracts the velocity
information from radar data, Cross- and Self-Attention Module (CSAM), to fuse
multi-scale radar features, and Pose Refinement Graph Convolutional Networks
(PRGCN), to refine the predicted keypoint confidence heatmaps. Our intensive
experiments on the HuPR benchmark show that the proposed scheme achieves better
human pose estimation performance with only radar data, as compared to
traditional pre-processing solutions and previous radio-frequency-based
methods.
