  The rapid growth of memory and computation requirements of large language
models (LLMs) has outpaced the development of hardware, hindering people who
lack large-scale high-end GPUs from training or deploying LLMs. However,
consumer-level GPUs, which constitute a larger market share, are typically
overlooked in LLM due to their weaker computing performance, smaller storage
capacity, and lower communication bandwidth. Additionally, users may have
privacy concerns when interacting with remote LLMs. In this paper, we envision
a decentralized system unlocking the potential vast untapped consumer-level
GPUs in pre-training, inference and fine-tuning of LLMs with privacy
protection. However, this system faces critical challenges, including limited
CPU and GPU memory, low network bandwidth, the variability of peer and device
heterogeneity. To address these challenges, our system design incorporates: 1)
a broker with backup pool to implement dynamic join and quit of computing
providers; 2) task scheduling with hardware performance to improve system
efficiency; 3) abstracting ML procedures into directed acyclic graphs (DAGs) to
achieve model and task universality; 4) abstracting intermediate represention
and execution planes to ensure compatibility of various devices and deep
learning (DL) frameworks. Our performance analysis demonstrates that 50 RTX
3080 GPUs can achieve throughputs comparable to those of 4 H100 GPUs, which are
significantly more expensive.
