  When analyzing human motion videos, the output jitters from existing pose
estimators are highly-unbalanced with varied estimation errors across frames.
Most frames in a video are relatively easy to estimate and only suffer from
slight jitters. In contrast, for rarely seen or occluded actions, the estimated
positions of multiple joints largely deviate from the ground truth values for a
consecutive sequence of frames, rendering significant jitters on them. To
tackle this problem, we propose to attach a dedicated temporal-only refinement
network to existing pose estimators for jitter mitigation, named SmoothNet.
Unlike existing learning-based solutions that employ spatio-temporal models to
co-optimize per-frame precision and temporal smoothness at all the joints,
SmoothNet models the natural smoothness characteristics in body movements by
learning the long-range temporal relations of every joint without considering
the noisy correlations among joints. With a simple yet effective motion-aware
fully-connected network, SmoothNet improves the temporal smoothness of existing
pose estimators significantly and enhances the estimation accuracy of those
challenging frames as a side-effect. Moreover, as a temporal-only model, a
unique advantage of SmoothNet is its strong transferability across various
types of estimators and datasets. Comprehensive experiments on five datasets
with eleven popular backbone networks across 2D and 3D pose estimation and body
recovery tasks demonstrate the efficacy of the proposed solution. Code is
available at https://github.com/cure-lab/SmoothNet.
