  Architectures with multiple classes of memory media are becoming a common
part of mainstream supercomputer deployments. So called multi-level memories
offer differing characteristics for each memory component including variation
in bandwidth, latency and capacity. This paper investigates the performance of
sparse matrix multiplication kernels on two leading high-performance computing
architectures -- Intel's Knights Landing processor and NVIDIA's Pascal GPU. We
describe a data placement method and a chunking-based algorithm for our kernels
that exploits the existence of the multiple memory spaces in each hardware
platform. We evaluate the performance of these methods w.r.t. standard
algorithms using the auto-caching mechanisms. Our results show that standard
algorithms that exploit cache reuse performed as well as multi-memory-aware
algorithms for architectures such as KNLs where the memory subsystems have
similar latencies. However, for architectures such as GPUs where memory
subsystems differ significantly in both bandwidth and latency,
multi-memory-aware methods are crucial for good performance. In addition, our
new approaches permit the user to run problems that require larger capacities
than the fastest memory of each compute node without depending on the
software-managed cache mechanisms.
