  Vertical Federated Learning (VFL) attracts increasing attention because it
empowers multiple parties to jointly train a privacy-preserving model over
vertically partitioned data. Recent research has shown that applying
zeroth-order optimization (ZOO) has many advantages in building a practical VFL
algorithm. However, a vital problem with the ZOO-based VFL is its slow
convergence rate, which limits its application in handling modern large models.
To address this problem, we propose a cascaded hybrid optimization method in
VFL. In this method, the downstream models (clients) are trained with ZOO to
protect privacy and ensure that no internal information is shared. Meanwhile,
the upstream model (server) is updated with first-order optimization (FOO)
locally, which significantly improves the convergence rate, making it feasible
to train the large models without compromising privacy and security. We
theoretically prove that our VFL framework converges faster than the ZOO-based
VFL, as the convergence of our framework is not limited by the size of the
server model, making it effective for training large models with the major part
on the server. Extensive experiments demonstrate that our method achieves
faster convergence than the ZOO-based VFL framework, while maintaining an
equivalent level of privacy protection. Moreover, we show that the convergence
of our VFL is comparable to the unsafe FOO-based VFL baseline. Additionally, we
demonstrate that our method makes the training of a large model feasible.
