  Multi-frame human pose estimation has long been a compelling and fundamental
problem in computer vision. This task is challenging due to fast motion and
pose occlusion that frequently occur in videos. State-of-the-art methods strive
to incorporate additional visual evidences from neighboring frames (supporting
frames) to facilitate the pose estimation of the current frame (key frame). One
aspect that has been obviated so far, is the fact that current methods directly
aggregate unaligned contexts across frames. The spatial-misalignment between
pose features of the current frame and neighboring frames might lead to
unsatisfactory results. More importantly, existing approaches build upon the
straightforward pose estimation loss, which unfortunately cannot constrain the
network to fully leverage useful information from neighboring frames. To tackle
these problems, we present a novel hierarchical alignment framework, which
leverages coarse-to-fine deformations to progressively update a neighboring
frame to align with the current frame at the feature level. We further propose
to explicitly supervise the knowledge extraction from neighboring frames,
guaranteeing that useful complementary cues are extracted. To achieve this
goal, we theoretically analyzed the mutual information between the frames and
arrived at a loss that maximizes the task-relevant mutual information. These
allow us to rank No.1 in the Multi-frame Person Pose Estimation Challenge on
benchmark dataset PoseTrack2017, and obtain state-of-the-art performance on
benchmarks Sub-JHMDB and Pose-Track2018. Our code is released at
https://github. com/Pose-Group/FAMI-Pose, hoping that it will be useful to the
community.
