  In surgical skill assessment, Objective Structured Assessments of Technical
Skills (OSATS scores) and the Global Rating Scale (GRS) are established tools
for evaluating the performance of surgeons during training. These metrics,
coupled with feedback on their performance, enable surgeons to improve and
achieve standards of practice. Recent studies on the open-source dataset
JIGSAW, which contains both GRS and OSATS labels, have focused on regressing
GRS scores from kinematic signals, video data, or a combination of both. In
this paper, we argue that regressing the GRS score, a unitless value, by itself
is too restrictive, and variations throughout the surgical trial do not hold
significant clinical meaning. To address this gap, we developed a recurrent
transformer model that outputs the surgeon's performance throughout their
training session by relating the model's hidden states to five OSATS scores
derived from kinematic signals. These scores are averaged and aggregated to
produce a GRS prediction, enabling assessment of the model's performance
against the state-of-the-art (SOTA). We report Spearman's Correlation
Coefficient (SCC), demonstrating that our model outperforms SOTA models for all
tasks, except for Suturing under the leave-one-subject-out (LOSO) scheme (SCC
0.68-0.89), while achieving comparable performance for suturing and across
tasks under the leave-one-user-out (LOUO) scheme (SCC 0.45-0.68) and beating
SOTA for Needle Passing (0.69). We argue that relating final OSATS scores to
short instances throughout a surgeon's procedure is more clinically meaningful
than a single GRS score. This approach also allows us to translate quantitative
predictions into qualitative feedback, which is crucial for any automated
surgical skill assessment pipeline. A senior surgeon validated our model's
behaviour and agreed with the semi-supervised predictions 77 \% (p = 0.006) of
the time.
