  Local feature attribution methods are increasingly used to explain complex
machine learning models. However, current methods are limited because they are
extremely expensive to compute or are not capable of explaining a distributed
series of models where each model is owned by a separate institution. The
latter is particularly important because it often arises in finance where
explanations are mandated. Here, we present DeepSHAP, a tractable method to
propagate local feature attributions through complex series of models based on
a connection to the Shapley value. We evaluate DeepSHAP across biological,
health, and financial datasets to show that it provides equally salient
explanations an order of magnitude faster than existing model-agnostic
attribution techniques and demonstrate its use in an important distributed
series of models setting.
