  The increasing size of language models necessitates a thorough analysis
across multiple dimensions to assess trade-offs among crucial hardware metrics
such as latency, energy consumption, GPU memory usage, and performance.
Identifying optimal model configurations under specific hardware constraints is
becoming essential but remains challenging due to the computational load of
exhaustive training and evaluation on multiple devices. To address this, we
introduce HW-GPT-Bench, a hardware-aware benchmark that utilizes surrogate
predictions to approximate various hardware metrics across 13 devices of
architectures in the GPT-2 family, with architectures containing up to 774M
parameters. Our surrogates, via calibrated predictions and reliable uncertainty
estimates, faithfully model the heteroscedastic noise inherent in the energy
and latency measurements. To estimate perplexity, we employ weight-sharing
techniques from Neural Architecture Search (NAS), inheriting pretrained weights
from the largest GPT-2 model. Finally, we demonstrate the utility of
HW-GPT-Bench by simulating optimization trajectories of various multi-objective
optimization algorithms in just a few seconds.
