  As deep learning models become increasingly bigger and more complex, it is
critical to improve model training and inference efficiency. Though a variety
of highly optimized libraries and packages (known as DL kernels) have been
developed, it is tedious and time-consuming to figure out which kernel to use,
where to use, and how to use them correctly. To address this challenge, we
propose an Automated Deep learning OPTimization approach called Adopter. We
design a Domain-Specific Language (DSL) to represent DL model architectures and
leverage this DSL to specify model transformation rules required to integrate a
DL kernel into a model. Given the source code of a DL model and the
transformation rules for a set of kernels, Adopter first performs
inter-procedural analysis to identify and express the model architecture in our
DSL. Then, Adopter performs scope analysis and sub-sequence matching to
identify locations in the model architecture where the transformation rules can
be applied. Finally, Adopter proposes a synthesis-based code transformation
method to apply the transformation rule. We curated a benchmark with 199 models
from Hugging Face and a diverse set of DL kernels. We found that, compared to a
state-of-the-art automated code transformation technique, Adopter helps improve
the precision and recall by 3% and 56%, respectively. An in-depth analysis of 9
models revealed that on average, Adopter improved the training speed by 22.7%
while decreasing the GPU memory usage by 10.5%.
