  Federated learning is an important privacy-preserving multi-party learning
paradigm, involving collaborative learning with others and local updating on
private data. Model heterogeneity and catastrophic forgetting are two crucial
challenges, which greatly limit the applicability and generalizability. This
paper presents a novel FCCL+, federated correlation and similarity learning
with non-target distillation, facilitating the both intra-domain
discriminability and inter-domain generalization. For heterogeneity issue, we
leverage irrelevant unlabeled public data for communication between the
heterogeneous participants. We construct cross-correlation matrix and align
instance similarity distribution on both logits and feature levels, which
effectively overcomes the communication barrier and improves the generalizable
ability. For catastrophic forgetting in local updating stage, FCCL+ introduces
Federated Non Target Distillation, which retains inter-domain knowledge while
avoiding the optimization conflict issue, fulling distilling privileged
inter-domain information through depicting posterior classes relation.
Considering that there is no standard benchmark for evaluating existing
heterogeneous federated learning under the same setting, we present a
comprehensive benchmark with extensive representative methods under four domain
shift scenarios, supporting both heterogeneous and homogeneous federated
settings. Empirical results demonstrate the superiority of our method and the
efficiency of modules on various scenarios.
