  Explanatory systems make machine learning models more transparent. However,
they are often inconsistent. In order to quantify and isolate possible
scenarios leading to this discrepancy, this paper compares two explanatory
systems, SHAP and LIME, based on the correlation of their respective importance
scores using 14 machine learning models (7 regression and 7 classification) and
4 tabular datasets (2 regression and 2 classification). We make two novel
findings. Firstly, the magnitude of importance is not significant in
explanation consistency. The correlations between SHAP and LIME importance
scores for the most important features may or may not be more variable than the
correlation between SHAP and LIME importance scores averaged across all
features. Secondly, the similarity between SHAP and LIME importance scores
cannot predict model accuracy. In the process of our research, we construct an
open-source library, XAISuite, that unifies the process of training and
explaining models. Finally, this paper contributes a generalized framework to
better explain machine learning models and optimize their performance.
