  This work presents a new method for enhancing communication efficiency in
stochastic Federated Learning that trains over-parameterized random networks.
In this setting, a binary mask is optimized instead of the model weights, which
are kept fixed. The mask characterizes a sparse sub-network that is able to
generalize as good as a smaller target network. Importantly, sparse binary
masks are exchanged rather than the floating point weights in traditional
federated learning, reducing communication cost to at most 1 bit per parameter
(Bpp). We show that previous state of the art stochastic methods fail to find
sparse networks that can reduce the communication and storage overhead using
consistent loss objectives. To address this, we propose adding a regularization
term to local objectives that acts as a proxy of the transmitted masks entropy,
therefore encouraging sparser solutions by eliminating redundant features
across sub-networks. Extensive empirical experiments demonstrate significant
improvements in communication and memory efficiency of up to five magnitudes
compared to the literature, with minimal performance degradation in validation
accuracy in some instances
