  Federated Learning (FL) trains a collaborative machine learning model by
aggregating multiple privately trained clients' models over several training
rounds. Such a long, continuous action of model aggregations poses significant
challenges in reasoning about the origin and composition of such a global
model. Regardless of the quality of the global model or if it has a fault,
understanding the model's origin is equally important for debugging,
interpretability, and explainability in federated learning. FL application
developers often question: (1) what clients contributed towards a global model
and (2) if a global model predicts a label, which clients are responsible for
it?
  We introduce, neuron provenance, a fine-grained lineage capturing mechanism
that tracks the flow of information between the individual participating
clients in FL and the final global model. We operationalize this concept in
ProvFL that functions on two key principles. First, recognizing that monitoring
every neuron of every client's model statically is ineffective and noisy due to
the uninterpretable nature of individual neurons, ProvFL dynamically isolates
influential and sensitive neurons in the global model, significantly reducing
the search space. Second, as multiple clients' models are fused in each round
to form a global model, tracking each client's contribution becomes
challenging. ProvFL leverages the invertible nature of fusion algorithms to
precisely isolate each client's contribution derived from selected neurons.
When asked to localize the clients responsible for the given behavior (i.e.,
prediction) of the global model, ProvFL successfully localizes them with an
average provenance accuracy of 97%. Additionally, ProvFL outperforms the
state-of-the-art FL fault localization approach by an average margin of 50%.
