  Personalized Federated Learning (FL) is an emerging research field in FL that
learns an easily adaptable global model in the presence of data heterogeneity
among clients. However, one of the main challenges for personalized FL is the
heavy reliance on clients' computing resources to calculate higher-order
gradients since client data is segregated from the server to ensure privacy. To
resolve this, we focus on a problem setting where the server may possess its
own data independent of clients' data -- a prevalent problem setting in various
applications, yet relatively unexplored in existing literature. Specifically,
we propose FedSIM, a new method for personalized FL that actively utilizes such
server data to improve meta-gradient calculation in the server for increased
personalization performance. Experimentally, we demonstrate through various
benchmarks and ablations that FedSIM is superior to existing methods in terms
of accuracy, more computationally efficient by calculating the full
meta-gradients in the server, and converges up to 34.2% faster.
