  Serverless computing has seen rapid adoption due to its high scalability and
flexible, pay-as-you-go billing model. In serverless, developers structure
their services as a collection of functions, sporadically invoked by various
events like clicks. High inter-arrival time variability of function invocations
motivates the providers to start new function instances upon each invocation,
leading to significant cold-start delays that degrade user experience. To
reduce cold-start latency, the industry has turned to snapshotting, whereby an
image of a fully-booted function is stored on disk, enabling a faster
invocation compared to booting a function from scratch.
  This work introduces vHive, an open-source framework for serverless
experimentation with the goal of enabling researchers to study and innovate
across the entire serverless stack. Using vHive, we characterize a
state-of-the-art snapshot-based serverless infrastructure, based on
industry-leading Containerd orchestration framework and Firecracker hypervisor
technologies. We find that the execution time of a function started from a
snapshot is 95% higher, on average, than when the same function is
memory-resident. We show that the high latency is attributable to frequent page
faults as the function's state is brought from disk into guest memory one page
at a time. Our analysis further reveals that functions access the same stable
working set of pages across different invocations of the same function. By
leveraging this insight, we build REAP, a light-weight software mechanism for
serverless hosts that records functions' stable working set of guest memory
pages and proactively prefetches it from disk into memory. Compared to baseline
snapshotting, REAP slashes the cold-start delays by 3.7x, on average.
