  Human hands possess remarkable dexterity and have long served as a source of
inspiration for robotic manipulation. In this work, we propose a human
$\textbf{H}$and$\textbf{-In}$formed visual representation learning framework to
solve difficult $\textbf{Dex}$terous manipulation tasks ($\textbf{H-InDex}$)
with reinforcement learning. Our framework consists of three stages: (i)
pre-training representations with 3D human hand pose estimation, (ii) offline
adapting representations with self-supervised keypoint detection, and (iii)
reinforcement learning with exponential moving average BatchNorm. The last two
stages only modify $0.36\%$ parameters of the pre-trained representation in
total, ensuring the knowledge from pre-training is maintained to the full
extent. We empirically study 12 challenging dexterous manipulation tasks and
find that H-InDex largely surpasses strong baseline methods and the recent
visual foundation models for motor control. Code is available at
https://yanjieze.com/H-InDex .
