  In the Industrial Internet of Things (IoT), a large amount of data will be
generated every day. Due to privacy and security issues, it is difficult to
collect all these data together to train deep learning models, thus the
federated learning, a distributed machine learning paradigm that protects data
privacy, has been widely used in IoT. However, in practical federated learning,
the data distributions usually have large differences across devices, and the
heterogeneity of data will deteriorate the performance of the model. Moreover,
federated learning in IoT usually has a large number of devices involved in
training, and the limited communication resource of cloud servers become a
bottleneck for training. To address the above issues, in this paper, we combine
centralized federated learning with decentralized federated learning to design
a semi-decentralized cloud-edge-device hierarchical federated learning
framework, which can mitigate the impact of data heterogeneity, and can be
deployed at lage scale in IoT. To address the effect of data heterogeneity, we
use an incremental subgradient optimization algorithm in each ring cluster to
improve the generalization ability of the ring cluster models. Our extensive
experiments show that our approach can effectively mitigate the impact of data
heterogeneity and alleviate the communication bottleneck in cloud servers.
