  Sparse linear algebra kernels play a critical role in numerous applications,
covering from exascale scientific simulation to large-scale data analytics.
Offloading linear algebra kernels on one GPU will no longer be viable in these
applications, simply because the rapidly growing data volume may exceed the
memory capacity and computing power of a single GPU. Multi-GPU systems nowadays
being ubiquitous in supercomputers and data-centers present great potentials in
scaling up large sparse linear algebra kernels. In this work, we design a novel
sparse matrix representation framework for multi-GPU systems called MSREP, to
scale sparse linear algebra operations based on our augmented sparse matrix
formats in a balanced pattern. Different from dense operations, sparsity
significantly intensifies the difficulty of distributing the computation
workload among multiple GPUs in a balanced manner. We enhance three mainstream
sparse data formats -- CSR, CSC, and COO, to enable fine-grained data
distribution. We take sparse matrix-vector multiplication (SpMV) as an example
to demonstrate the efficiency of our MSREP framework. In addition, MSREP can be
easily extended to support other sparse linear algebra kernels based on the
three fundamental formats (i.e., CSR, CSC and COO).
