  Hardware platforms in high performance computing are constantly getting more
complex to handle even when considering multicore CPUs alone. Numerous features
and configuration options in the hardware and the software environment that are
relevant for performance are not even known to most application users or
developers. Microbenchmarks, i.e., simple codes that fathom a particular aspect
of the hardware, can help to shed light on such issues, but only if they are
well understood and if the results can be reconciled with known facts or
performance models. The insight gained from microbenchmarks may then be applied
to real applications for performance analysis or optimization. In this paper we
investigate two modern Intel x86 server CPU architectures in depth: Broadwell
EP and Cascade Lake SP. We highlight relevant hardware configuration settings
that can have a decisive impact on code performance and show how to properly
measure on-chip and off-chip data transfer bandwidths. The new victim L3 cache
of Cascade Lake and its advanced replacement policy receive due attention.
Finally we use DGEMM, sparse matrix-vector multiplication, and the HPCG
benchmark to make a connection to relevant application scenarios.
