  The development of Large Language Models (LLMs) faces a significant
challenge: the exhausting of publicly available fresh data. This is because
training a LLM needs a large demanding of new data. Federated learning emerges
as a promising solution, enabling collaborative model to contribute their
private data to LLM global model. However, integrating federated learning with
LLMs introduces new challenges, including the lack of transparency and the need
for effective unlearning mechanisms. Transparency is essential to ensuring
trust and fairness among participants, while accountability is crucial for
deterring malicious behaviour and enabling corrective actions when necessary.
To address these challenges, we propose a novel blockchain-based federated
learning framework for LLMs that enhances transparency, accountability, and
unlearning capabilities. Our framework leverages blockchain technology to
create a tamper-proof record of each model's contributions and introduces an
innovative unlearning function that seamlessly integrates with the federated
learning mechanism. We investigate the impact of Low-Rank Adaptation (LoRA)
hyperparameters on unlearning performance and integrate Hyperledger Fabric to
ensure the security, transparency, and verifiability of the unlearning process.
Through comprehensive experiments and analysis, we showcase the effectiveness
of our proposed framework in achieving highly effective unlearning in LLMs
trained using federated learning. Our findings highlight the feasibility of
integrating blockchain technology into federated learning frameworks for LLMs.
