  Interpreting predictions from tree ensemble methods such as gradient boosting
machines and random forests is important, yet feature attribution for trees is
often heuristic and not individualized for each prediction. Here we show that
popular feature attribution methods are inconsistent, meaning they can lower a
feature's assigned importance when the true impact of that feature actually
increases. This is a fundamental problem that casts doubt on any comparison
between features. To address it we turn to recent applications of game theory
and develop fast exact tree solutions for SHAP (SHapley Additive exPlanation)
values, which are the unique consistent and locally accurate attribution
values. We then extend SHAP values to interaction effects and define SHAP
interaction values. We propose a rich visualization of individualized feature
attributions that improves over classic attribution summaries and partial
dependence plots, and a unique "supervised" clustering (clustering based on
feature attributions). We demonstrate better agreement with human intuition
through a user study, exponential improvements in run time, improved clustering
performance, and better identification of influential features. An
implementation of our algorithm has also been merged into XGBoost and LightGBM,
see http://github.com/slundberg/shap for details.
