  Deep Learning (DL) has developed to become a corner-stone in many everyday
applications that we are now relying on. However, making sure that the DL model
uses the underlying hardware efficiently takes a lot of effort. Knowledge about
inference characteristics can help to find the right match so that enough
resources are given to the model, but not too much. We have developed a DL
Inference Performance Predictive Model (DIPPM) that predicts the inference
latency, energy, and memory usage of a given input DL model on the NVIDIA A100
GPU. We also devised an algorithm to suggest the appropriate A100
Multi-Instance GPU profile from the output of DIPPM. We developed a methodology
to convert DL models expressed in multiple frameworks to a generalized graph
structure that is used in DIPPM. It means DIPPM can parse input DL models from
various frameworks. Our DIPPM can be used not only helps to find suitable
hardware configurations but also helps to perform rapid design-space
exploration for the inference performance of a model. We constructed a graph
multi-regression dataset consisting of 10,508 different DL models to train and
evaluate the performance of DIPPM, and reached a resulting Mean Absolute
Percentage Error (MAPE) as low as 1.9%.
