  In this work, fundamental performance, power, and energy characteristics of
the full SPEChpc 2021 benchmark suite are assessed on two different clusters
based on Intel Ice Lake and Sapphire Rapids CPUs using the MPI-only codes'
variants. We use memory bandwidth, data volume, and scalability metrics in
order to categorize the benchmarks and pinpoint relevant performance and
scalability bottlenecks on the node and cluster levels. Common patterns such as
memory bandwidth limitation, dominating communication and synchronization
overhead, MPI serialization, superlinear scaling, and alignment issues could be
identified, in isolation or in combination, showing that SPEChpc 2021 is
representative of many HPC workloads. Power dissipation and energy measurements
indicate that the modern Intel server CPUs have such a high idle power level
that race-to-idle is the paramount strategy for energy to solution and
energy-delay product minimization. On the chip level, only memory-bound code
shows a clear advantage of Sapphire Rapids compared to Ice Lake in terms of
energy to solution.
