  As Internet of Things (IoT) technology advances, end devices like sensors and
smartphones are progressively equipped with AI models tailored to their local
memory and computational constraints. Local inference reduces communication
costs and latency; however, these smaller models typically underperform
compared to more sophisticated models deployed on edge servers or in the cloud.
Cooperative Inference Systems (CISs) address this performance trade-off by
enabling smaller devices to offload part of their inference tasks to more
capable devices. These systems often deploy hierarchical models that share
numerous parameters, exemplified by Deep Neural Networks (DNNs) that utilize
strategies like early exits or ordered dropout. In such instances, Federated
Learning (FL) may be employed to jointly train the models within a CIS. Yet,
traditional training methods have overlooked the operational dynamics of CISs
during inference, particularly the potential high heterogeneity in serving
rates across clients. To address this gap, we propose a novel FL approach
designed explicitly for use in CISs that accounts for these variations in
serving rates. Our framework not only offers rigorous theoretical guarantees,
but also surpasses state-of-the-art (SOTA) training algorithms for CISs,
especially in scenarios where inference request rates or data availability are
uneven among clients.
