  Neural Architecture Search (NAS) for Federated Learning (FL) is an emerging
field. It automates the design and training of Deep Neural Networks (DNNs) when
data cannot be centralized due to privacy, communication costs, or regulatory
restrictions. Recent federated NAS methods not only reduce manual effort but
also help achieve higher accuracy than traditional FL methods like FedAvg.
Despite the success, existing federated NAS methods still fall short in
satisfying diverse deployment targets common in on-device inference like
hardware, latency budgets, or variable battery levels. Most federated NAS
methods search for only a limited range of neuro-architectural patterns, repeat
them in a DNN, thereby restricting achievable performance. Moreover, these
methods incur prohibitive training costs to satisfy deployment targets. They
perform the training and search of DNN architectures repeatedly for each case.
SuperFedNAS addresses these challenges by decoupling the training and search in
federated NAS. SuperFedNAS co-trains a large number of diverse DNN
architectures contained inside one supernet in the FL setting. Post-training,
clients perform NAS locally to find specialized DNNs by extracting different
parts of the trained supernet with no additional training. SuperFedNAS takes
O(1) (instead of O(N)) cost to find specialized DNN architectures in FL for any
N deployment targets. As part of SuperFedNAS, we introduce MaxNet - a novel FL
training algorithm that performs multi-objective federated optimization of a
large number of DNN architectures ($\approx 5*10^8$) under different client
data distributions. Overall, SuperFedNAS achieves upto 37.7% higher accuracy
for the same MACs or upto 8.13x reduction in MACs for the same accuracy than
existing federated NAS methods.
