  With growing concerns regarding data privacy and rapid increase in data
volume, Federated Learning(FL) has become an important learning paradigm.
However, jointly learning a deep neural network model in a FL setting proves to
be a non-trivial task because of the complexities associated with the neural
networks, such as varied architectures across clients, permutation invariance
of the neurons, and presence of non-linear transformations in each layer. This
work introduces a novel Federated Heterogeneous Neural Networks (FedHeNN)
framework that allows each client to build a personalised model without
enforcing a common architecture across clients. This allows each client to
optimize with respect to local data and compute constraints, while still
benefiting from the learnings of other (potentially more powerful) clients. The
key idea of FedHeNN is to use the instance-level representations obtained from
peer clients to guide the simultaneous training on each client. The extensive
experimental results demonstrate that the FedHeNN framework is capable of
learning better performing models on clients in both the settings of
homogeneous and heterogeneous architectures across clients.
