  We propose Flexible Vertical Federated Learning (Flex-VFL), a distributed
machine algorithm that trains a smooth, non-convex function in a distributed
system with vertically partitioned data. We consider a system with several
parties that wish to collaboratively learn a global function. Each party holds
a local dataset; the datasets have different features but share the same sample
ID space. The parties are heterogeneous in nature: the parties' operating
speeds, local model architectures, and optimizers may be different from one
another and, further, they may change over time. To train a global model in
such a system, Flex-VFL utilizes a form of parallel block coordinate descent,
where parties train a partition of the global model via stochastic coordinate
descent. We provide theoretical convergence analysis for Flex-VFL and show that
the convergence rate is constrained by the party speeds and local optimizer
parameters. We apply this analysis and extend our algorithm to adapt party
learning rates in response to changing speeds and local optimizer parameters.
Finally, we compare the convergence time of Flex-VFL against synchronous and
asynchronous VFL algorithms, as well as illustrate the effectiveness of our
adaptive extension.
