  Federated Learning (FL) is a machine learning approach where nodes
collaboratively train a global model. As more nodes participate in a round of
FL, the effectiveness of individual model updates by nodes also diminishes. In
this study, we increase the effectiveness of client updates by dividing the
network into smaller partitions, or cohorts. We introduce Cohort-Parallel
Federated Learning (CPFL): a novel learning approach where each cohort
independently trains a global model using FL, until convergence, and the
produced models by each cohort are then unified using one-shot Knowledge
Distillation (KD) and a cross-domain, unlabeled dataset. The insight behind
CPFL is that smaller, isolated networks converge quicker than in a one-network
setting where all nodes participate. Through exhaustive experiments involving
realistic traces and non-IID data distributions on the CIFAR-10 and FEMNIST
image classification tasks, we investigate the balance between the number of
cohorts, model accuracy, training time, and compute and communication
resources. Compared to traditional FL, CPFL with four cohorts, non-IID data
distribution, and CIFAR-10 yields a 1.9$\times$ reduction in train time and a
1.3$\times$ reduction in resource usage, with a minimal drop in test accuracy.
