  Semantic segmentation in surgical videos has applications in intra-operative
guidance, post-operative analytics and surgical education. Segmentation models
need to provide accurate and consistent predictions since temporally
inconsistent identification of anatomical structures can impair usability and
hinder patient safety. Video information can alleviate these challenges leading
to reliable models suitable for clinical use. We propose a novel architecture
for modelling temporal relationships in videos. The proposed model includes a
spatio-temporal decoder to enable video semantic segmentation by improving
temporal consistency across frames. The encoder processes individual frames
whilst the decoder processes a temporal batch of adjacent frames. The proposed
decoder can be used on top of any segmentation encoder to improve temporal
consistency. Model performance was evaluated on the CholecSeg8k dataset and a
private dataset of robotic Partial Nephrectomy procedures. Segmentation
performance was improved when the temporal decoder was applied across both
datasets. The proposed model also displayed improvements in temporal
consistency.
