  Purpose: General consensus amongst researchers and industry points to a lack
of large, representative annotated datasets as the biggest obstacle to progress
in the field of surgical data science. Self-supervised learning represents a
solution to part of this problem, removing the reliance on annotations.
However, the robustness of current self-supervised learning methods to domain
shifts remains unclear, limiting our understanding of its utility for
leveraging diverse sources of surgical data. Methods: In this work, we employ
self-supervised learning to flexibly leverage diverse surgical datasets,
thereby learning taskagnostic representations that can be used for various
surgical downstream tasks. Based on this approach, to elucidate the impact of
pre-training on downstream task performance, we explore 22 different
pre-training dataset combinations by modulating three variables: source
hospital, type of surgical procedure, and pre-training scale (number of
videos). We then finetune the resulting model initializations on three diverse
downstream tasks: namely, phase recognition and critical view of safety in
laparoscopic cholecystectomy and phase recognition in laparoscopic
hysterectomy. Results: Controlled experimentation highlights sizable boosts in
performance across various tasks, datasets, and labeling budgets. However, this
performance is intricately linked to the composition of the pre-training
dataset, robustly proven through several study stages. Conclusion: The
composition of pre-training datasets can severely affect the effectiveness of
SSL methods for various downstream tasks and should critically inform future
data collection efforts to scale the application of SSL methodologies.
  Keywords: Self-Supervised Learning, Transfer Learning, Surgical Computer
Vision, Endoscopic Videos, Critical View of Safety, Phase Recognition
