  Image-based tracking of medical instruments is an integral part of surgical
data science applications. Previous research has addressed the tasks of
detecting, segmenting and tracking medical instruments based on laparoscopic
video data. However, the proposed methods still tend to fail when applied to
challenging images and do not generalize well to data they have not been
trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set -
the first publicly available data set enabling comprehensive benchmarking of
medical instrument detection and segmentation algorithms with a specific
emphasis on method robustness and generalization capabilities. Our data set
comprises 30 laparoscopic videos and corresponding sensor data from medical
devices in the operating room for three different types of laparoscopic
surgery. Annotations include surgical phase labels for all video frames as well
as information on instrument presence and corresponding instance-wise
segmentation masks for surgical instruments (if any) in more than 10,000
individual frames. The data has successfully been used to organize
international competitions within the Endoscopic Vision Challenges 2017 and
2019.
