  While GPUs are responsible for training the vast majority of state-of-the-art
deep learning models, the implications of their architecture are often
overlooked when designing new deep learning (DL) models. As a consequence,
modifying a DL model to be more amenable to the target hardware can
significantly improve the runtime performance of DL training and inference. In
this paper, we provide a set of guidelines for users to maximize the runtime
performance of their transformer models. These guidelines have been created by
carefully considering the impact of various model hyperparameters controlling
model shape on the efficiency of the underlying computation kernels executed on
the GPU. We find the throughput of models with efficient model shapes is up to
39\% higher while preserving accuracy compared to models with a similar number
of parameters but with unoptimized shapes.
