  As surgical robots become more common, automating away some of the burden of
complex direct human operation becomes ever more feasible. Model-free
reinforcement learning (RL) is a promising direction toward generalizable
automated surgical performance, but progress has been slowed by the lack of
efficient and realistic learning environments. In this paper, we describe
adding reinforcement learning support to the da Vinci Skill Simulator, a
training simulation used around the world to allow surgeons to learn and
rehearse technical skills. We successfully teach an RL-based agent to perform
sub-tasks in the simulator environment, using either image or state data. As
far as we know, this is the first time an RL-based agent is taught from visual
data in a surgical robotics environment. Additionally, we tackle the sample
inefficiency of RL using a simple-to-implement system which we term
hybrid-batch learning (HBL), effectively adding a second, long-term replay
buffer to the Q-learning process. Additionally, this allows us to bootstrap
learning from images from the data collected using the easier task of learning
from state. We show that HBL decreases our learning times significantly.
