  This work deals with the CPU-GPU heterogeneous code acceleration of a
finite-volume CFD solver utilizing multiple CPUs and GPUs at the same time.
First, a high-level description of the CFD solver called SENSEI, the
discretization of SENSEI, and the CPU-GPU heterogeneous computing workflow in
SENSEI leveraging MPI and OpenACC are given. Then, a performance model for
CPU-GPU heterogeneous computing requiring ghost cell exchange is proposed to
help estimate the performance of the heterogeneous implementation. The scaling
performance of the CPU-GPU heterogeneous computing and its comparison with the
pure multi-CPU/GPU performance for a supersonic inlet test case is presented to
display the advantages of leveraging the computational power of both the CPU
and the GPU. Using CPUs and GPUs as workers together, the performance can be
improved further compared to using pure CPUs or GPUs, and the advantages can be
fairly estimated by the performance model proposed in this work. Finally,
conclusions are drawn to provide 1) suggestions for application users who have
an interest to leverage the computational power of the CPU and GPU to
accelerate their own scientific computing simulations and 2) feedback for
hardware architects who have an interest to design a better CPU-GPU
heterogeneous system for heterogeneous computing.
