  Federated Learning is introduced to protect privacy by distributing training
data into multiple parties. Each party trains its own model and a meta-model is
constructed from the sub models. In this way the details of the data are not
disclosed in between each party. In this paper we investigate the model
interpretation methods for Federated Learning, specifically on the measurement
of feature importance of vertical Federated Learning where feature space of the
data is divided into two parties, namely host and guest. For host party to
interpret a single prediction of vertical Federated Learning model, the
interpretation results, namely the feature importance, are very likely to
reveal the protected data from guest party. We propose a method to balance the
model interpretability and data privacy in vertical Federated Learning by using
Shapley values to reveal detailed feature importance for host features and a
unified importance value for federated guest features. Our experiments indicate
robust and informative results for interpreting Federated Learning models.
