  The plethora of complex artificial intelligence (AI) algorithms and available
high performance computing (HPC) power stimulates the expeditious development
of AI components with heterogeneous designs. Consequently, the need for
cross-stack performance benchmarking of AI-HPC systems emerges rapidly. The de
facto HPC benchmark LINPACK can not reflect AI computing power and I/O
performance without representative workload. The current popular AI benchmarks
like MLPerf have fixed problem size therefore limited scalability. To address
these issues, we propose an end-to-end benchmark suite utilizing automated
machine learning (AutoML), which not only represents real AI scenarios, but
also is auto-adaptively scalable to various scales of machines. We implement
the algorithms in a highly parallel and flexible way to ensure the efficiency
and optimization potential on diverse systems with customizable configurations.
We utilize operations per second (OPS), which is measured in an analytical and
systematic approach, as the major metric to quantify the AI performance. We
perform evaluations on various systems to ensure the benchmark's stability and
scalability, from 4 nodes with 32 NVIDIA Tesla T4 (56.1 Tera-OPS measured), up
to 512 nodes with 4096 Huawei Ascend 910 (194.53 Peta-OPS measured), and the
results show near-linear weak scalability. With flexible workload and single
metric, our benchmark can scale and rank AI-HPC easily.
