  The A64FX CPU is arguably the most powerful Arm-based processor design to
date. Although it is a traditional cache-based multicore processor, its peak
performance and memory bandwidth rival accelerator devices. A good
understanding of its performance features is of paramount importance for
developers who wish to leverage its full potential. We present an architectural
analysis of the A64FX used in the Fujitsu FX1000 supercomputer at a level of
detail that allows for the construction of Execution-Cache-Memory (ECM)
performance models for steady-state loops. In the process we identify
architectural peculiarities that point to viable generic optimization
strategies. After validating the model using simple streaming loops we apply
the insight gained to sparse matrix-vector multiplication (SpMV) and the domain
wall (DW) kernel from quantum chromodynamics (QCD). For SpMV we show why the
CRS matrix storage format is not a good practical choice on this architecture
and how the SELL-C-sigma format can achieve bandwidth saturation. For the DW
kernel we provide a cache-reuse analysis and show how an appropriate choice of
data layout for complex arrays can realize memory-bandwidth saturation in this
case as well. A comparison with state-of-the-art high-end Intel Cascade Lake AP
and Nvidia V100 systems puts the capabilities of the A64FX into perspective. We
also explore the potential for power optimizations using the tuning knobs
provided by the Fugaku system, achieving energy savings of about 31% for SpMV
and 18% for DW.
