  Federated learning (FL) allows multiple medical institutions to
collaboratively learn a global model without centralizing client data. It is
difficult, if possible at all, for such a global model to commonly achieve
optimal performance for each individual client, due to the heterogeneity of
medical images from various scanners and patient demographics. This problem
becomes even more significant when deploying the global model to unseen clients
outside the FL with unseen distributions not presented during federated
training. To optimize the prediction accuracy of each individual client for
medical imaging tasks, we propose a novel unified framework for both
\textit{Inside and Outside model Personalization in FL} (IOP-FL). Our inside
personalization uses a lightweight gradient-based approach that exploits the
local adapted model for each client, by accumulating both the global gradients
for common knowledge and the local gradients for client-specific optimization.
Moreover, and importantly, the obtained local personalized models and the
global model can form a diverse and informative routing space to personalize an
adapted model for outside FL clients. Hence, we design a new test-time routing
scheme using the consistency loss with a shape constraint to dynamically
incorporate the models, given the distribution information conveyed by the test
data. Our extensive experimental results on two medical image segmentation
tasks present significant improvements over SOTA methods on both inside and
outside personalization, demonstrating the potential of our IOP-FL scheme for
clinical practice.
