  We developed a deep learning classifier of rectal cancer response (tumor vs.
no-tumor) to total neoadjuvant treatment (TNT) from endoscopic images acquired
before, during, and following TNT. We further evaluated the network's ability
in a near out-of-distribution (OOD) problem to identify local regrowth (LR)
from follow-up endoscopy images acquired several months to years after
completing TNT. We addressed endoscopic image variability by using optimal mass
transport-based image harmonization. We evaluated multiple training
regularization schemes to study the ResNet-50 network's in-distribution and
near-OOD generalization ability. Test time augmentation resulted in the most
considerable accuracy improvement. Image harmonization resulted in slight
accuracy improvement for the near-OOD cases. Our results suggest that
off-the-shelf deep learning classifiers can detect rectal cancer from
endoscopic images at various stages of therapy for surveillance.
