  We present SurgeonAssist-Net: a lightweight framework making
action-and-workflow-driven virtual assistance, for a set of predefined surgical
tasks, accessible to commercially available optical see-through head-mounted
displays (OST-HMDs). On a widely used benchmark dataset for laparoscopic
surgical workflow, our implementation competes with state-of-the-art approaches
in prediction accuracy for automated task recognition, and yet requires 7.4x
fewer parameters, 10.2x fewer floating point operations per second (FLOPS), is
7.0x faster for inference on a CPU, and is capable of near real-time
performance on the Microsoft HoloLens 2 OST-HMD. To achieve this, we make use
of an efficient convolutional neural network (CNN) backbone to extract
discriminative features from image data, and a low-parameter recurrent neural
network (RNN) architecture to learn long-term temporal dependencies. To
demonstrate the feasibility of our approach for inference on the HoloLens 2 we
created a sample dataset that included video of several surgical tasks recorded
from a user-centric point-of-view. After training, we deployed our model and
cataloged its performance in an online simulated surgical scenario for the
prediction of the current surgical task. The utility of our approach is
explored in the discussion of several relevant clinical use-cases. Our code is
publicly available at https://github.com/doughtmw/surgeon-assist-net.
