  Recent advancements in surgical computer vision have been driven by
vision-only models, which lack language semantics, relying on manually
annotated videos to predict fixed object categories. This limits their
generalizability to unseen surgical procedures and tasks. We propose leveraging
surgical video lectures from e-learning platforms to provide effective vision
and language supervisory signals for multi-modal representation learning,
bypassing manual annotations. We address surgery-specific linguistic challenges
using multiple automatic speech recognition systems for text transcriptions. We
introduce SurgVLP - Surgical Vision Language Pre-training - a novel method for
multi-modal representation learning. SurgVLP employs a new contrastive learning
objective, aligning video clip embeddings with corresponding multiple text
embeddings in a joint latent space. We demonstrate the representational
capability of this space through several vision-and-language surgical tasks and
vision-only tasks specific to surgery. Unlike current fully supervised
approaches, SurgVLP adapts to different surgical procedures and tasks without
specific fine-tuning, achieving zero-shot adaptation to tasks such as surgical
tool, phase, and triplet recognition without manual annotation. These results
highlight the transferability and versatility of the learned multi-modal
representations in surgical video analysis. The code is available at
https://github.com/CAMMA-public/SurgVLP
