  Surgical captioning plays an important role in surgical instruction
prediction and report generation. However, the majority of captioning models
still rely on the heavy computational object detector or feature extractor to
extract regional features. In addition, the detection model requires additional
bounding box annotation which is costly and needs skilled annotators. These
lead to inference delay and limit the captioning model to deploy in real-time
robotic surgery. For this purpose, we design an end-to-end detector and feature
extractor-free captioning model by utilizing the patch-based shifted window
technique. We propose Shifted Window-Based Multi-Layer Perceptrons Transformer
Captioning model (SwinMLP-TranCAP) with faster inference speed and less
computation. SwinMLP-TranCAP replaces the multi-head attention module with
window-based multi-head MLP. Such deployments primarily focus on image
understanding tasks, but very few works investigate the caption generation
task. SwinMLP-TranCAP is also extended into a video version for video
captioning tasks using 3D patches and windows. Compared with previous
detector-based or feature extractor-based models, our models greatly simplify
the architecture design while maintaining performance on two surgical datasets.
The code is publicly available at
https://github.com/XuMengyaAmy/SwinMLP_TranCAP.
