  Shapley Values, a solution to the credit assignment problem in cooperative
game theory, are a popular type of explanation in machine learning, having been
used to explain the importance of features, embeddings, and even neurons. In
NLP, however, leave-one-out and attention-based explanations still predominate.
Can we draw a connection between these different methods? We formally prove
that -- save for the degenerate case -- attention weights and leave-one-out
values cannot be Shapley Values. $\textit{Attention flow}$ is a post-processed
variant of attention weights obtained by running the max-flow algorithm on the
attention graph. Perhaps surprisingly, we prove that attention flows are indeed
Shapley Values, at least at the layerwise level. Given the many desirable
theoretical qualities of Shapley Values -- which has driven their adoption
among the ML community -- we argue that NLP practitioners should, when
possible, adopt attention flow explanations alongside more traditional ones.
