  Enable neural networks to capture 3D geometrical-aware features is essential
in multi-view based vision tasks. Previous methods usually encode the 3D
information of multi-view stereo into the 2D features. In contrast, we present
a novel method, named POEM, that directly operates on the 3D POints Embedded in
the Multi-view stereo for reconstructing hand mesh in it. Point is a natural
form of 3D information and an ideal medium for fusing features across views, as
it has different projections on different views. Our method is thus in light of
a simple yet effective idea, that a complex 3D hand mesh can be represented by
a set of 3D points that 1) are embedded in the multi-view stereo, 2) carry
features from the multi-view images, and 3) encircle the hand. To leverage the
power of points, we design two operations: point-based feature fusion and
cross-set point attention mechanism. Evaluation on three challenging multi-view
datasets shows that POEM outperforms the state-of-the-art in hand mesh
reconstruction. Code and models are available for research at
https://github.com/lixiny/POEM.
