  An obvious way to alleviate memory difficulties in GPU-based AI computing is
via CPU offload, where data are moved between GPU and CPU RAM, so inexpensive
CPU RAM is used to increase the amount of storage available. While CPU offload
is an obvious idea, it can greatly slow down a computation, due to the
relatively slow transfer rate between CPU RAM and GPU RAM. Thus, any system for
CPU offload needs to ensure that when such a transfer needs to happen, no
computation is blocked waiting for the transfer to finish. One of the key
challenges when using CPU offload is that memory transfers introduce
nondeterminacy into the system: it is not possible to know before runtime when
the transfers will finish, and hence what is the best order of operations to
run to ensure there is no blocking. In this paper, we describe TURNIP, which is
a system for running AI computations using CPU offload. The key innovation in
TURNIP is the compilation of the AI computation into a dependency graph that
gives the TURNIP runtime freedom to run operations such as GPU kernel calls in
many different orders; at runtime, TURNIP chooses the best order in response to
real-time events.
