  Federated Learning (FL) is a recent model training paradigm in which client
devices collaboratively train a model without ever aggregating their data.
Crucially, this scheme offers users potential privacy and security benefits by
only ever communicating updates to the model weights to a central server as
opposed to traditional machine learning (ML) training which directly
communicates and aggregates data. However, FL training suffers from statistical
heterogeneity as clients may have differing local data distributions. Large
language models (LLMs) offer a potential solution to this issue of
heterogeneity given that they have consistently been shown to be able to learn
on vast amounts of noisy data. While LLMs are a promising development for
resolving the consistent issue of non-I.I.D. Clients in federated settings
exacerbate two other bottlenecks in FL: limited local computing and expensive
communication. This thesis aims to develop efficient training methods for LLMs
in FL. To this end, we employ two critical techniques in enabling efficient
training. First, we use low-rank adaptation (LoRA) to reduce the computational
load of local model training. Second, we communicate sparse updates throughout
training to significantly cut down on communication costs. Taken together, our
method reduces communication costs by up to 10x over vanilla LoRA and up to 5x
over more complex sparse LoRA baselines while achieving greater utility. We
emphasize the importance of carefully applying sparsity and picking effective
rank and sparsity configurations for federated LLM training.
