  Training deep learning (DL) models in the cloud has become a norm. With the
emergence of serverless computing and its benefits of true pay-as-you-go
pricing and scalability, systems researchers have recently started to provide
support for serverless-based training. However, the ability to train DL models
on serverless platforms is hindered by the resource limitations of today's
serverless infrastructure and DL models' explosive requirement for memory and
bandwidth. This paper describes FuncPipe, a novel pipelined training framework
specifically designed for serverless platforms that enable fast and low-cost
training of DL models. FuncPipe is designed with the key insight that model
partitioning can be leveraged to bridge both memory and bandwidth gaps between
the capacity of serverless functions and the requirement of DL training.
Conceptually simple, we have to answer several design questions, including how
to partition the model, configure each serverless function, and exploit each
function's uplink/downlink bandwidth. In particular, we tailor a micro-batch
scheduling policy for the serverless environment, which serves as the basis for
the subsequent optimization. Our Mixed-Integer Quadratic Programming
formulation automatically and simultaneously configures serverless resources
and partitions models to fit within the resource constraints. Lastly, we
improve the bandwidth efficiency of storage-based synchronization with a novel
pipelined scatter-reduce algorithm. We implement FuncPipe on two popular cloud
serverless platforms and show that it achieves 7%-77% cost savings and
1.3X-2.2X speedup compared to state-of-the-art serverless-based frameworks.
