  Recent research advance has significantly improved the visual realism of
immersive 3D video communication. In this work we present a method to further
enhance this immersive experience by adding the hand touch capability ("remote
hand clapping"). In our system, each meeting participant sits in front of a
large screen with haptic feedback. The local participant can reach his hand out
to the screen and perform hand clapping with the remote participant as if the
two participants were only separated by a virtual glass. A key challenge in
emulating the remote hand touch is the realistic rendering of the participant's
hand and arm as the hand touches the screen. When the hand is very close to the
screen, the RGBD data required for realistic rendering is no longer available.
To tackle this challenge, we present a dual representation of the user's hand.
Our dual representation not only preserves the high-quality rendering usually
found in recent image-based rendering systems but also allows the hand to reach
the screen. This is possible because the dual representation includes both an
image-based model and a 3D geometry-based model, with the latter driven by a
hand skeleton tracked by a side view camera. In addition, the dual
representation provides a distance-based fusion of the image-based and 3D
geometry-based models as the hand moves closer to the screen. The result is
that the image-based and 3D geometry-based models mutually enhance each other,
leading to realistic and seamless rendering. Our experiments demonstrate that
our method provides consistent hand contact experience between remote users and
improves the immersive experience of 3D video communication.
