  Basic Linear Algebra Subprograms (BLAS) are a set of low level linear algebra
kernels widely adopted by applications involved with the deep learning and
scientific computing. The massive and economic computing power brought forth by
the emerging GPU architectures drives interest in implementation of
compute-intensive level 3 BLAS on multi-GPU systems. In this paper, we
investigate existing multi-GPU level 3 BLAS and present that 1) issues, such as
the improper load balancing, inefficient communication, insufficient GPU stream
level concurrency and data caching, impede current implementations from fully
harnessing heterogeneous computing resources; 2) and the inter-GPU
Peer-to-Peer(P2P) communication remains unexplored. We then present BLASX: a
highly optimized multi-GPU level-3 BLAS. We adopt the concepts of
algorithms-by-tiles treating a matrix tile as the basic data unit and
operations on tiles as the basic task. Tasks are guided with a dynamic
asynchronous runtime, which is cache and locality aware. The communication cost
under BLASX becomes trivial as it perfectly overlaps communication and
computation across multiple streams during asynchronous task progression. It
also takes the current tile cache scheme one step further by proposing an
innovative 2-level hierarchical tile cache, taking advantage of inter-GPU P2P
communication. As a result, linear speedup is observable with BLASX under
multi-GPU configurations; and the extensive benchmarks demonstrate that BLASX
consistently outperforms the related leading industrial and academic projects
such as cuBLAS-XT, SuperMatrix, MAGMA and PaRSEC.
