  Traditional datasets for the radiological diagnosis tend to only provide the
radiology image alongside the radiology report. However, radiology reading as
performed by radiologists is a complex process, and information such as the
radiologist's eye-fixations over the course of the reading has the potential to
be an invaluable data source to learn from. Nonetheless, the collection of such
data is expensive and time-consuming. This leads to the question of whether
such data is worth the investment to collect. This paper utilizes the recently
published Eye-Gaze dataset to perform an exhaustive study on the impact on
performance and explainability of deep learning (DL) classification in the face
of varying levels of input features, namely: radiology images, radiology report
text, and radiologist eye-gaze data. We find that the best classification
performance of X-ray images is achieved with a combination of radiology report
free-text and radiology image, with the eye-gaze data providing no performance
boost. Nonetheless, eye-gaze data serving as secondary ground truth alongside
the class label results in highly explainable models that generate better
attention maps compared to models trained to do classification and attention
map generation without eye-gaze data.
