  Fine-grained activity recognition enables explainable analysis of procedures
for skill assessment, autonomy, and error detection in robot-assisted surgery.
However, existing recognition models suffer from the limited availability of
annotated datasets with both kinematic and video data and an inability to
generalize to unseen subjects and tasks. Kinematic data from the surgical robot
is particularly critical for safety monitoring and autonomy, as it is
unaffected by common camera issues such as occlusions and lens contamination.
We leverage an aggregated dataset of six dry-lab surgical tasks from a total of
28 subjects to train activity recognition models at the gesture and motion
primitive (MP) levels and for separate robotic arms using only kinematic data.
The models are evaluated using the LOUO (Leave-One-User-Out) and our proposed
LOTO (Leave-One-Task-Out) cross validation methods to assess their ability to
generalize to unseen users and tasks respectively. Gesture recognition models
achieve higher accuracies and edit scores than MP recognition models. But,
using MPs enables the training of models that can generalize better to unseen
tasks. Also, higher MP recognition accuracy can be achieved by training
separate models for the left and right robot arms. For task-generalization, MP
recognition models perform best if trained on similar tasks and/or tasks from
the same dataset.
