  NVIDIA Tensor Core is a mixed-precision matrix-matrix multiplication and
addition computing unit, where the theoretical peak performance is more than
300 TFlop/s on NVIDIA A100 GPU. NVIDIA provides WMMA API for using Tensor Cores
in custom kernel functions. The most common way to use Tensor Core is to supply
the input matrices from shared memory, which has higher bandwidth than global
memory. However, the Bytes-per-Flops (B/F) ratio of the shared memory and
Tensor Cores is small since the performance of Tensor Cores is high. Thus, it
is important to reduce the shared memory footprint for efficient Tensor Cores
usage. In this paper, we analyze the simple matrix-matrix multiplication on
Tensor Cores by the roofline model and figure out that the bandwidth of shared
memory might be a limitation of the performance when using WMMA API. To
alleviate this issue, we provide a WMMA API extension library to boost the
throughput of the computation, which has two components. The first one allows
for manipulating the array of registers input to Tensor Cores flexibly. We
evaluate the performance improvement of this library. The outcome of our
evaluation shows that our library reduces the shared memory footprint and
speeds up the computation using Tensor Cores. The second one is an API for the
SGEMM emulation on Tensor Cores without additional shared memory usage. We have
demonstrated that the single-precision emulating batch SGEMM implementation on
Tensor Cores using this library achieves 54.2 TFlop/s on A100 GPU, which
outperforms the theoretical peak performance of FP32 SIMT Cores while achieving
the same level of accuracy as cuBLAS. The achieved throughput can not be
achieved without reducing the shared memory footprint done by our library with
the same amount of register usage.
