  Explainable AI (XAI) has a counterpart in analytical modeling which we refer
to as model explainability. We tackle the issue of model explainability in the
context of prediction models. We analyze a dataset of loans from a credit card
company and apply three stages: execute and compare four different prediction
methods, apply the best known explainability techniques in the current
literature to the model training sets to identify feature importance (FI)
(static case), and finally to cross-check whether the FI set holds up under
what if prediction scenarios for continuous and categorical variables (dynamic
case). We found inconsistency in FI identification between the static and
dynamic cases. We summarize the state of the art in model explainability and
suggest further research to advance the field.
