  Detecting lesions from computed tomography (CT) scans is an important but
difficult problem because non-lesions and true lesions can appear similar. 3D
context is known to be helpful in this differentiation task. However, existing
end-to-end detection frameworks of convolutional neural networks (CNNs) are
mostly designed for 2D images. In this paper, we propose 3D context enhanced
region-based CNN (3DCE) to incorporate 3D context information efficiently by
aggregating feature maps of 2D images. 3DCE is easy to train and end-to-end in
training and inference. A universal lesion detector is developed to detect all
kinds of lesions in one algorithm using the DeepLesion dataset. Experimental
results on this challenging task prove the effectiveness of 3DCE. We have
released the code of 3DCE in
https://github.com/rsummers11/CADLab/tree/master/lesion_detector_3DCE.
