  Accurately estimating 3D hand pose is crucial for understanding how humans
interact with the world. Despite remarkable progress, existing methods often
struggle to generate plausible hand poses when the hand is heavily occluded or
blurred. In videos, the movements of the hand allow us to observe various parts
of the hand that may be occluded or blurred in a single frame. To adaptively
leverage the visual clue before and after the occlusion or blurring for robust
hand pose estimation, we propose the Deformer: a framework that implicitly
reasons about the relationship between hand parts within the same image
(spatial dimension) and different timesteps (temporal dimension). We show that
a naive application of the transformer self-attention mechanism is not
sufficient because motion blur or occlusions in certain frames can lead to
heavily distorted hand features and generate imprecise keys and queries. To
address this challenge, we incorporate a Dynamic Fusion Module into Deformer,
which predicts the deformation of the hand and warps the hand mesh predictions
from nearby frames to explicitly support the current frame estimation.
Furthermore, we have observed that errors are unevenly distributed across
different hand parts, with vertices around fingertips having disproportionately
higher errors than those around the palm. We mitigate this issue by introducing
a new loss function called maxMSE that automatically adjusts the weight of
every vertex to focus the model on critical hand parts. Extensive experiments
show that our method significantly outperforms state-of-the-art methods by 10%,
and is more robust to occlusions (over 14%).
