  Semantic segmentation and activity classification are key components to
creating intelligent surgical systems able to understand and assist clinical
workflow. In the Operating Room, semantic segmentation is at the core of
creating robots aware of clinical surroundings, whereas activity classification
aims at understanding OR workflow at a higher level. State-of-the-art semantic
segmentation and activity recognition approaches are fully supervised, which is
not scalable. Self-supervision can decrease the amount of annotated data
needed. We propose a new 3D self-supervised task for OR scene understanding
utilizing OR scene images captured with ToF cameras. Contrary to other
self-supervised approaches, where handcrafted pretext tasks are focused on 2D
image features, our proposed task consists of predicting the relative 3D
distance of image patches by exploiting the depth maps. Learning 3D spatial
context generates discriminative features for our downstream tasks. Our
approach is evaluated on two tasks and datasets containing multi-view data
captured from clinical scenarios. We demonstrate a noteworthy improvement of
performance on both tasks, specifically on low-regime data where utility of
self-supervised learning is the highest.
