  The ability to model, analyze, and predict execution time of computations is
an important building block supporting numerous efforts, such as load
balancing, performance optimization, and automated performance tuning for high
performance, parallel applications. In today's increasingly heterogeneous
computing environment, this task must be accomplished efficiently across
multiple architectures, including massively parallel coprocessors like GPUs. To
address this challenge, we present an approach for constructing customizable,
cross-machine performance models for GPU kernels, including a mechanism to
automatically and symbolically gather performance-relevant kernel operation
counts, a tool for formulating mathematical models using these counts, and a
customizable parameterized collection of benchmark kernels used to calibrate
models to GPUs in a black-box fashion. Our approach empowers a user to manage
trade-offs between model accuracy, evaluation speed, and generalizability. A
user can define a model and customize the calibration process, making it as
simple or complex as desired, and as application-targeted or general as
desired. To evaluate our approach, we demonstrate both linear and nonlinear
models; each example models execution times for multiple variants of a
particular computation: two matrix multiplication variants, four Discontinuous
Galerkin (DG) differentiation operation variants, and two 2-D five-point finite
difference stencil variants. For each variant, we present accuracy results on
GPUs from multiple vendors and hardware generations. We view this customizable
approach as a response to a central question in GPU performance modeling: how
can we model GPU performance in a cost-explanatory fashion while maintaining
accuracy, evaluation speed, portability, and ease of use, an attribute we
believe precludes manual collection of kernel or hardware statistics.
