  In treating gastrointestinal cancer using radiotherapy, the role of the
radiation oncologist is to administer high doses of radiation, through x-ray
beams, toward the tumor while avoiding the stomach and intestines. With the
advent of precise radiation treatment technology such as the MR-Linac,
oncologists can visualize the daily positions of the tumors and intestines,
which may vary day to day. Before delivering radiation, radio oncologists must
manually outline the position of the gastrointestinal organs in order to
determine position and direction of the x-ray beam. This is a time consuming
and labor intensive process that may substantially prolong a patient's
treatment. A deep learning (DL) method can automate and expedite the process.
However, many deep neural networks approaches currently in use are black-boxes
which lack interpretability which render them untrustworthy and impractical in
a healthcare setting. To address this, an emergent field of AI known as
Explainable AI (XAI) may be incorporated to improve the transparency and
viability of a model. This paper proposes a deep learning pipeline that
incorporates XAI to address the challenges of organ segmentation.
