  The emergence of large-scale foundation models (FoMo's) that can perform
human-like intelligence motivates their deployment at the network edge for
devices to access state-of-the-art artificial intelligence. For better user
experiences, the pre-trained FoMo's need to be adapted to specialized
downstream tasks through fine-tuning techniques. To transcend a single device's
memory and computation limitations, we advocate multi-device cooperation within
the device-edge cooperative fine-tuning (DEFT) paradigm, where edge devices
cooperate to simultaneously optimize different parts of fine-tuning parameters
within a FoMo. However, the parameter blocks reside at different depths within
a FoMo architecture, leading to varied computation latency-and-memory cost due
to gradient backpropagation-based calculations. The heterogeneous on-device
computation and memory capacities and channel conditions necessitate an
integrated communication-and-computation allocation of local computation loads
and communication resources to achieve low-latency (LoLa) DEFT. To this end, we
consider the depth-ware DEFT block allocation problem. The involved optimal
block-device matching is tackled by the proposed low-complexity
Cutting-RecoUNting-CHecking (CRUNCH) algorithm, which is designed by exploiting
the monotone-increasing property between block depth and computation
latency-and-memory cost. Next, the joint bandwidth-and-block allocation makes
the problem more sophisticated. We observe a splittable Lagrangian expression
through the transformation and analysis of the original problem, where the
variables indicating device involvement are introduced. Then, the dual ascent
method is employed to tackle this problem iteratively. Through extensive
experiments conducted on the GLUE benchmark, our results demonstrate
significant latency reduction achievable by LoLa DEFT for fine-tuning a RoBERTa
model.
