  With the growing adoption of AI-based systems across everyday life, the need
to understand their decision-making mechanisms is correspondingly increasing.
The level at which we can trust the statistical inferences made from AI-based
decision systems is an increasing concern, especially in high-risk systems such
as criminal justice or medical diagnosis, where incorrect inferences may have
tragic consequences. Despite their successes in providing solutions to problems
involving real-world data, deep learning (DL) models cannot quantify the
certainty of their predictions. These models are frequently quite confident,
even when their solutions are incorrect.
  This work presents a method to infer prominent features in two DL
classification models trained on clinical and non-clinical text by employing
techniques from topological and geometric data analysis. We create a graph of a
model's feature space and cluster the inputs into the graph's vertices by the
similarity of features and prediction statistics. We then extract subgraphs
demonstrating high-predictive accuracy for a given label. These subgraphs
contain a wealth of information about features that the DL model has recognized
as relevant to its decisions. We infer these features for a given label using a
distance metric between probability measures, and demonstrate the stability of
our method compared to the LIME and SHAP interpretability methods. This work
establishes that we may gain insights into the decision mechanism of a DL
model. This method allows us to ascertain if the model is making its decisions
based on information germane to the problem or identifies extraneous patterns
within the data.
