  Adverse surgical outcomes are costly to patients and hospitals. Approaches to
benchmark surgical care are often limited to gross measures across the entire
procedure despite the performance of particular tasks being largely responsible
for undesirable outcomes. In order to produce metrics from tasks as opposed to
the whole procedure, methods to recognize automatically individual surgical
tasks are needed. In this paper, we propose several approaches to recognize
surgical activities in robot-assisted minimally invasive surgery using deep
learning. We collected a clinical dataset of 100 robot-assisted radical
prostatectomies (RARP) with 12 tasks each and propose `RP-Net', a modified
version of InceptionV3 model, for image based surgical activity recognition. We
achieve an average precision of 80.9% and average recall of 76.7% across all
tasks using RP-Net which out-performs all other RNN and CNN based models
explored in this paper. Our results suggest that automatic surgical activity
recognition during RARP is feasible and can be the foundation for advanced
analytics.
