  In real-world federated learning scenarios, participants could have their own
personalized labels which are incompatible with those from other clients, due
to using different label permutations or tackling completely different tasks or
domains. However, most existing FL approaches cannot effectively tackle such
extremely heterogeneous scenarios since they often assume that (1) all
participants use a synchronized set of labels, and (2) they train on the same
task from the same domain. In this work, to tackle these challenges, we
introduce Factorized-FL, which allows to effectively tackle label- and
task-heterogeneous federated learning settings by factorizing the model
parameters into a pair of vectors, where one captures the common knowledge
across different labels and tasks and the other captures knowledge specific to
the task each local model tackles. Moreover, based on the distance in the
client-specific vector space, Factorized-FL performs selective aggregation
scheme to utilize only the knowledge from the relevant participants for each
client. We extensively validate our method on both label- and
domain-heterogeneous settings, on which it outperforms the state-of-the-art
personalized federated learning methods.
