  To develop intelligent speech assistants and integrate them seamlessly with
intra-operative decision-support frameworks, accurate and efficient surgical
phase recognition is a prerequisite. In this study, we propose a multimodal
framework based on Gated Multimodal Units (GMU) and Multi-Stage Temporal
Convolutional Networks (MS-TCN) to recognize surgical phases of port-catheter
placement operations. Our method merges speech and image models and uses them
separately in different surgical phases. Based on the evaluation of 28
operations, we report a frame-wise accuracy of 92.65 $\pm$ 3.52% and an
F1-score of 92.30 $\pm$ 3.82%. Our results show approximately 10% improvement
in both metrics over previous work and validate the effectiveness of
integrating multimodal data for the surgical phase recognition task. We further
investigate the contribution of individual data channels by comparing
mono-modal models with multimodal models.
