  While FPGA accelerator boards and their respective high-level design tools
are maturing, there is still a lack of multi-FPGA applications, libraries, and
not least, benchmarks and reference implementations towards sustained HPC usage
of these devices. As in the early days of GPUs in HPC, for workloads that can
reasonably be decoupled into loosely coupled working sets, multi-accelerator
support can be achieved by using standard communication interfaces like MPI on
the host side. However, for performance and productivity, some applications can
profit from a tighter coupling of the accelerators. FPGAs offer unique
opportunities here when extending the dataflow characteristics to their
communication ininterfaces. In this work, we extend the HPCC FPGA benchmark
suite by multi-FPGA support and three missing benchmarks that particularly
characterize or stress inter-device communication: b_eff, PTRANS, and LINPACK.
With all benchmarks implemented for current boards with Intel and Xilinx FPGAs,
we established a baseline for multi-FPGA performance. Additionally, for the
communication-centric benchmarks, we explored the potential of direct
FPGA-to-FPGA communication with a circuit-switched inter-FPGA network that is
currently only available for one of the boards. The evaluation with parallel
execution on up to 26 FPGA boards makes use of one of the largest academic FPGA
installations.
