  Interpretability is a pressing issue for decision systems. Many post hoc
methods have been proposed to explain the predictions of a single machine
learning model. However, business processes and decision systems are rarely
centered around a unique model. These systems combine multiple models that
produce key predictions, and then apply decision rules to generate the final
decision. To explain such decisions, we propose the Semi-Model-Agnostic
Contextual Explainer (SMACE), a new interpretability method that combines a
geometric approach for decision rules with existing interpretability methods
for machine learning models to generate an intuitive feature ranking tailored
to the end user. We show that established model-agnostic approaches produce
poor results on tabular data in this setting, in particular giving the same
importance to several features, whereas SMACE can rank them in a meaningful
way.
