  The GPU programming model is primarily aimed at the development of
applications that run one GPU. However, this limits the scalability of GPU code
to the capabilities of a single GPU in terms of compute power and memory
capacity. To scale GPU applications further, a great engineering effort is
typically required: work and data must be divided over multiple GPUs by hand,
possibly in multiple nodes, and data must be manually spilled from GPU memory
to higher-level memories.
  We present Lightning: a framework that follows the common GPU programming
paradigm but enables scaling to large problems with ease. Lightning supports
multi-GPU execution of GPU kernels, even across multiple nodes, and seamlessly
spills data to higher-level memories (main memory and disk). Existing CUDA
kernels can easily be adapted for use in Lightning, with data access
annotations on these kernels allowing Lightning to infer their data
requirements and the dependencies between subsequent kernel launches. Lightning
efficiently distributes the work/data across GPUs and maximizes efficiency by
overlapping scheduling, data movement, and kernel execution when possible.
  We present the design and implementation of Lightning, as well as
experimental results on up to 32 GPUs for eight benchmarks and one real-world
application. Evaluation shows excellent performance and scalability, such as a
speedup of 57.2x over the CPU using Lighting with 16 GPUs over 4 nodes and 80
GB of data, far beyond the memory capacity of one GPU.
