  Massively multi-task learning with large language models has recently made
substantial progress on few-shot generalization. However, this is usually
performed in a centralized learning fashion, ignoring the privacy sensitivity
issue of (annotated) data used in multiple tasks. To mitigate this issue, we
propose FewFedWeight, a few-shot federated learning framework across multiple
tasks, to achieve the best of both worlds: privacy preservation and cross-task
generalization. FewFedWeight trains client models in isolated devices without
sharing data. It broadcasts the global model in the server to each client and
produces pseudo data for clients so that knowledge from the global model can be
explored to enhance few-shot learning of each client model. An energy-based
algorithm is further proposed to weight pseudo samples in order to reduce the
negative impact of noise from the generated pseudo data. Adaptive model weights
of client models are also tuned according to their performance. We use these
model weights to dynamically aggregate client models to update the global
model. Experiments on 118 NLP tasks show that FewFedWeight can significantly
improve the performance of client models on 61% tasks with an average
performance improvement rate of 30.5% over the baseline and substantially
outperform FedAvg and other decentralized learning methods.
