  Modern heterogeneous supercomputing systems are comprised of compute blades
that offer CPUs and GPUs. On such systems, it is essential to move data
efficiently between these different compute engines across a high-speed
network. While current generation scientific applications and systems software
stacks are GPU-aware, CPU threads are still required to orchestrate data moving
communication operations and inter-process synchronization operations.
  A new GPU stream-aware MPI communication strategy called stream-triggered
(ST) communication is explored to allow offloading both computation and
communication control paths to the GPU. The proposed ST communication strategy
is implemented on HPE Slingshot Interconnects over a new proprietary HPE
Slingshot NIC (Slingshot 11) using the supported triggered operations feature.
Performance of the proposed new communication strategy is evaluated using a
microbenchmark kernel called Faces, based on the nearest-neighbor communication
pattern in the CORAL-2 Nekbone benchmark, over a heterogeneous node
architecture consisting of AMD CPUs and GPUs.
