  The never-ending computational demand from simulations of turbulence makes
computational fluid dynamics (CFD) a prime application use case for current and
future exascale systems. High-order finite element methods, such as the
spectral element method, have been gaining traction as they offer high
performance on both multicore CPUs and modern GPU-based accelerators. In this
work, we assess how high-fidelity CFD using the spectral element method can
exploit the modular supercomputing architecture at scale through domain
partitioning, where the computational domain is split between a Booster module
powered by GPUs and a Cluster module with conventional CPU nodes. We
investigate several different flow cases and computer systems based on the
modular supercomputing architecture (MSA). We observe that for our simulations,
the communication overhead and load balancing issues incurred by incorporating
different computing architectures are seldom worthwhile, especially when I/O is
also considered, but when the simulation at hand requires more than the
combined global memory on the GPUs, utilizing additional CPUs to increase the
available memory can be fruitful. We support our results with a simple
performance model to assess when running across modules might be beneficial. As
MSA is becoming more widespread and efforts to increase system utilization are
growing more important our results give insight into when and how a monolithic
application can utilize and spread out to more than one module and obtain a
faster time to solution.
