  The prediction of solar flares is of practical and scientific interest;
however, many machine learning methods used for this prediction task do not
provide the physical explanations behind a model's performance. We made use of
two recently developed explainable artificial intelligence techniques called
gradient-weighted class activation mapping (Grad-CAM) and expected gradients
(EG) to reveal the decision-making process behind a high-performance neural
network that has been trained to distinguish between MgII spectra derived from
flaring and nonflaring active regions, a fact that can be applied to the task
of short timescale flare forecasting. The two techniques generate visual
explanations (heatmaps) that can be projected back onto the spectra, allowing
for the identification of features that are strongly associated with precursory
flare activity. We automated the search for explainable interpretations on the
level of individual wavelengths, and provide multiple examples of flare
prediction using IRIS spectral data, finding that prediction scores in general
increase before flare onset. Large IRIS rasters that cover a significant
portion of the active region and coincide with small preflare brightenings both
in IRIS and SDO/AIA images tend to lead to better forecasts. The models reveal
that MgII triplet emission, flows, as well as broad and highly asymmetric
spectra are all important for the task of flare prediction. Additionally, we
find that intensity is only weakly correlated to a spectrum's prediction score,
meaning that low intensity spectra can still be of great importance for the
flare prediction task, and that $78$% of the time, the position of the model's
maximum attention along the slit during the preflare phase is predictive of the
location of the flare's maximum UV emission
