  Surgical activity recognition and prediction can help provide important
context in many Robot-Assisted Surgery (RAS) applications, for example,
surgical progress monitoring and estimation, surgical skill evaluation, and
shared control strategies during teleoperation. Transformer models were first
developed for Natural Language Processing (NLP) to model word sequences and
soon the method gained popularity for general sequence modeling tasks. In this
paper, we propose the novel use of a Transformer model for three tasks: gesture
recognition, gesture prediction, and trajectory prediction during RAS. We
modify the original Transformer architecture to be able to generate the current
gesture sequence, future gesture sequence, and future trajectory sequence
estimations using only the current kinematic data of the surgical robot
end-effectors. We evaluate our proposed models on the JHU-ISI Gesture and Skill
Assessment Working Set (JIGSAWS) and use Leave-One-User-Out (LOUO)
cross-validation to ensure the generalizability of our results. Our models
achieve up to 89.3\% gesture recognition accuracy, 84.6\% gesture prediction
accuracy (1 second ahead) and 2.71mm trajectory prediction error (1 second
ahead). Our models are comparable to and able to outperform state-of-the-art
methods while using only the kinematic data channel. This approach can enable
near-real time surgical activity recognition and prediction.
