  To investigate the heterogeneity in federated learning in real-world
scenarios, we generalize the classic federated learning to federated
hetero-task learning, which emphasizes the inconsistency across the
participants in federated learning in terms of both data distribution and
learning tasks. We also present B-FHTL, a federated hetero-task learning
benchmark consisting of simulation dataset, FL protocols and a unified
evaluation mechanism. B-FHTL dataset contains three well-designed federated
learning tasks with increasing heterogeneity. Each task simulates the clients
with different non-IID data and learning tasks. To ensure fair comparison among
different FL algorithms, B-FHTL builds in a full suite of FL protocols by
providing high-level APIs to avoid privacy leakage, and presets most common
evaluation metrics spanning across different learning tasks, such as
regression, classification, text generation and etc. Furthermore, we compare
the FL algorithms in fields of federated multi-task learning, federated
personalization and federated meta learning within B-FHTL, and highlight the
influence of heterogeneity and difficulties of federated hetero-task learning.
Our benchmark, including the federated dataset, protocols, the evaluation
mechanism and the preliminary experiment, is open-sourced at
https://github.com/alibaba/FederatedScope/tree/master/benchmark/B-FHTL
