  Dexterous telemanipulation is crucial in advancing human-robot systems,
especially in tasks requiring precise and safe manipulation. However, it faces
significant challenges due to the physical differences between human and
robotic hands, the dynamic interaction with objects, and the indirect control
and perception of the remote environment. Current approaches predominantly
focus on mapping the human hand onto robotic counterparts to replicate motions,
which exhibits a critical oversight: it often neglects the physical interaction
with objects and relegates the interaction burden to the human to adapt and
make laborious adjustments in response to the indirect and counter-intuitive
observation of the remote environment. This work develops an
End-Effects-Oriented Learning-based Dexterous Telemanipulation (EFOLD)
framework to address telemanipulation tasks. EFOLD models telemanipulation as a
Markov Game, introducing multiple end-effect features to interpret the human
operator's commands during interaction with objects. These features are used by
a Deep Reinforcement Learning policy to control the robot and reproduce such
end effects. EFOLD was evaluated with real human subjects and two end-effect
extraction methods for controlling a virtual Shadow Robot Hand in
telemanipulation tasks. EFOLD achieved real-time control capability with low
command following latency (delay<0.11s) and highly accurate tracking (MSE<0.084
rad).
