  Histopathological characterization of colorectal polyps is an important
principle for determining the risk of colorectal cancer and future rates of
surveillance for patients. This characterization is time-intensive, requires
years of specialized training, and suffers from significant inter-observer and
intra-observer variability. In this work, we built an automatic
image-understanding method that can accurately classify different types of
colorectal polyps in whole-slide histology images to help pathologists with
histopathological characterization and diagnosis of colorectal polyps. The
proposed image-understanding method is based on deep-learning techniques, which
rely on numerous levels of abstraction for data representation and have shown
state-of-the-art results for various image analysis tasks. Our
image-understanding method covers all five polyp types (hyperplastic polyp,
sessile serrated polyp, traditional serrated adenoma, tubular adenoma, and
tubulovillous/villous adenoma) that are included in the US multi-society task
force guidelines for colorectal cancer risk assessment and surveillance, and
encompasses the most common occurrences of colorectal polyps. Our evaluation on
239 independent test samples shows our proposed method can identify the types
of colorectal polyps in whole-slide images with a high efficacy (accuracy:
93.0%, precision: 89.7%, recall: 88.3%, F1 score: 88.8%). The presented method
in this paper can reduce the cognitive burden on pathologists and improve their
accuracy and efficiency in histopathological characterization of colorectal
polyps, and in subsequent risk assessment and follow-up recommendations.
