  Limited training data and severe class imbalance impose significant
challenges to developing clinically robust deep learning models. Federated
learning (FL) addresses the former by enabling different medical clients to
collaboratively train a deep model without sharing data. However, the class
imbalance problem persists due to inter-client class distribution variations.
To overcome this, we propose federated classifier anchoring (FCA) by adding a
personalized classifier at each client to guide and debias the federated model
through consistency learning. Additionally, FCA debiases the federated
classifier and each client's personalized classifier based on their respective
class distributions, thus mitigating divergence. With FCA, the federated
feature extractor effectively learns discriminative features suitably globally
for federation as well as locally for all participants. In clinical practice,
the federated model is expected to be both generalized, performing well across
clients, and specialized, benefiting each individual client from collaboration.
According to this, we propose a novel evaluation metric to assess models'
generalization and specialization performance globally on an aggregated public
test set and locally at each client. Through comprehensive comparison and
evaluation, FCA outperforms the state-of-the-art methods with large margins for
federated long-tailed skin lesion classification and intracranial hemorrhage
classification, making it a more feasible solution in clinical settings.
