  This paper reports on the design and implementation of the HPC performance
monitoring system deployed to continuously monitor performance metrics of all
jobs on the HPC systems at the Max Planck Computing and Data Facility (MPCDF).
Thereby it reveals important information to various stakeholders, in particular
to users, application support, system administrators, and management. On each
compute node, hardware and software performance monitoring data is collected by
our newly developed lightweight open-source hpcmd middleware which builds upon
standard Linux tools. The data is transported via rsyslog, and aggregated and
processed by a Splunk system, enabling detailed per-cluster and per-job
interactive analysis in a web browser. Additionally, performance reports are
provided to the users as PDF files. Finally, we report on practical experience
and benefits from large-scale deployments on MPCDF HPC systems, demonstrating
how our solution can be useful to any HPC center.
