  Task automation of surgical robot has the potentials to improve surgical
efficiency. Recent reinforcement learning (RL) based approaches provide
scalable solutions to surgical automation, but typically require extensive data
collection to solve a task if no prior knowledge is given. This issue is known
as the exploration challenge, which can be alleviated by providing expert
demonstrations to an RL agent. Yet, how to make effective use of demonstration
data to improve exploration efficiency still remains an open challenge. In this
work, we introduce Demonstration-guided EXploration (DEX), an efficient
reinforcement learning algorithm that aims to overcome the exploration problem
with expert demonstrations for surgical automation. To effectively exploit
demonstrations, our method estimates expert-like behaviors with higher values
to facilitate productive interactions, and adopts non-parametric regression to
enable such guidance at states unobserved in demonstration data. Extensive
experiments on $10$ surgical manipulation tasks from SurRoL, a comprehensive
surgical simulation platform, demonstrate significant improvements in the
exploration efficiency and task success rates of our method. Moreover, we also
deploy the learned policies to the da Vinci Research Kit (dVRK) platform to
show the effectiveness on the real robot. Code is available at
https://github.com/med-air/DEX.
