  Machine Learning (ML) models execute several parallel computations including
Generalized Matrix Multiplication, Convolution, Dropout, etc. These
computations are commonly executed on Graphics Processing Units (GPUs), by
dividing the computation into independent processing blocks, known as tiles.
Since the number of tiles are usually higher than the execution units of a GPU,
tiles are executed on all execution units in one or more waves. However, the
number of tiles is not always a multiple of the number of execution units.
Thus, tiles executed in the final wave can under-utilize the GPU.
  To address this issue, we present cuSync, a framework for synchronizing
dependent kernels using a user-defined fine-grained synchronization policy to
improve the GPU utilization. cuSync synchronizes tiles instead of kernels,
which allows executing independent tiles of dependent kernels concurrently. We
also present a compiler to generate diverse fine-grained synchronization
policies based on dependencies between kernels. Our experiments found that
synchronizing CUDA kernels using cuSync reduces the inference times of four
popular ML models: MegatronLM GPT-3 by up to 15%, LLaMA by up to 14%, ResNet-38
by up to 22%, and VGG-19 by up to 16% over several batch sizes.
