  Federated learning (FL) enables distributed resource-constrained devices to
jointly train shared models while keeping the training data local for privacy
purposes. Vertical FL (VFL), which allows each client to collect partial
features, has attracted intensive research efforts recently. We identified the
main challenges that existing VFL frameworks are facing: the server needs to
communicate gradients with the clients for each training step, incurring high
communication cost that leads to rapid consumption of privacy budgets. To
address these challenges, in this paper, we introduce a VFL framework with
multiple heads (VIM), which takes the separate contribution of each client into
account, and enables an efficient decomposition of the VFL optimization
objective to sub-objectives that can be iteratively tackled by the server and
the clients on their own. In particular, we propose an Alternating Direction
Method of Multipliers (ADMM)-based method to solve our optimization problem,
which allows clients to conduct multiple local updates before communication,
and thus reduces the communication cost and leads to better performance under
differential privacy (DP). We provide the user-level DP mechanism for our
framework to protect user privacy. Moreover, we show that a byproduct of VIM is
that the weights of learned heads reflect the importance of local clients. We
conduct extensive evaluations and show that on four vertical FL datasets, VIM
achieves significantly higher performance and faster convergence compared with
the state-of-the-art. We also explicitly evaluate the importance of local
clients and show that VIM enables functionalities such as client-level
explanation and client denoising. We hope this work will shed light on a new
way of effective VFL training and understanding.
