  We present a study using a class of post-hoc local explanation methods i.e.,
feature importance methods for "understanding" a deep learning (DL) emulator of
climate. Specifically, we consider a multiple-input-single-output emulator that
uses a DenseNet encoder-decoder architecture and is trained to predict
interannual variations of sea surface temperature (SST) at 1, 6, and 9 month
lead times using the preceding 36 months of (appropriately filtered) SST data.
First, feature importance methods are employed for individual predictions to
spatio-temporally identify input features that are important for model
prediction at chosen geographical regions and chosen prediction lead times. In
a second step, we also examine the behavior of feature importance in a
generalized sense by considering an aggregation of the importance heatmaps over
training samples. We find that: 1) the climate emulator's prediction at any
geographical location depends dominantly on a small neighborhood around it; 2)
the longer the prediction lead time, the further back the "importance" extends;
and 3) to leading order, the temporal decay of "importance" is independent of
geographical location. An ablation experiment is adopted to verify the
findings. From the perspective of climate dynamics, these findings suggest a
dominant role for local processes and a negligible role for remote
teleconnections at the spatial and temporal scales we consider. From the
perspective of network architecture, the spatio-temporal relations between the
inputs and outputs we find suggest potential model refinements. We discuss
further extensions of our methods, some of which we are considering in ongoing
work.
