  Rationale: Computer aided detection (CAD) algorithms for Pulmonary Embolism
(PE) algorithms have been shown to increase radiologists' sensitivity with a
small increase in specificity. However, CAD for PE has not been adopted into
clinical practice, likely because of the high number of false positives current
CAD software produces. Objective: To generate a database of annotated computed
tomography pulmonary angiographies, use it to compare the sensitivity and false
positive rate of current algorithms and to develop new methods that improve
such metrics. Methods: 91 Computed tomography pulmonary angiography scans were
annotated by at least one radiologist by segmenting all pulmonary emboli
visible on the study. 20 annotated CTPAs were open to the public in the form of
a medical image analysis challenge. 20 more were kept for evaluation purposes.
51 were made available post-challenge. 8 submissions, 6 of them novel, were
evaluated on the 20 evaluation CTPAs. Performance was measured as per embolus
sensitivity vs. false positives per scan curve. Results: The best algorithms
achieved a per-embolus sensitivity of 75% at 2 false positives per scan (fps)
or of 70% at 1 fps, outperforming the state of the art. Deep learning
approaches outperformed traditional machine learning ones, and their
performance improved with the number of training cases. Significance: Through
this work and challenge we have improved the state-of-the art of computer aided
detection algorithms for pulmonary embolism. An open database and an evaluation
benchmark for such algorithms have been generated, easing the development of
further improvements. Implications on clinical practice will need further
research.
