  Transformer-based pre-trained models have emerged as the predominant solution
for natural language processing (NLP). Fine-tuning such pre-trained models for
downstream tasks often requires a considerable amount of labeled private data.
In practice, private data is often distributed across heterogeneous mobile
devices and may be prohibited from being uploaded. Moreover, well-curated
labeled data is often scarce, presenting an additional challenge. To address
these challenges, we first introduce a data generator for federated few-shot
learning tasks, which encompasses the quantity and skewness of scarce labeled
data in a realistic setting. Subsequently, we propose AUG-FedPrompt, a
prompt-based federated learning system that exploits abundant unlabeled data
for data augmentation. Our experiments indicate that AUG-FedPrompt can perform
on par with full-set fine-tuning with a limited amount of labeled data.
However, such competitive performance comes at a significant system cost.
