  Performance optimization is the art of continuous seeking a harmonious
mapping between the application domain and hardware. Recent years have
witnessed a surge of deep learning (DL) applications in industry. Conventional
wisdom for optimizing such workloads mainly focus on compute intensive ops
(GEMM, Convolution, etc). Yet we show in this work, that the performance of
memory intensive computations is vital to E2E performance in practical DL
models. We propose \emph{FusionStitching}, a optimization framework capable of
fusing memory intensive \emph{elementwise}, \emph{reduction} and fine grained
\emph{GEMM/Batched-GEMM} ops, with or without data dependences, into large
computation units, then mapping and transforming them into efficient GPU
kernels. We formulate the fusion plan optimization as an integer linear
programming (ILP) problem, and propose a set of empirical heuristics to reduce
the combinatorial search space. In order to map optimized fusion plans to
hardware, we propose a technique to effectively compose various groups of
computations into a single GPU kernel, by fully leveraging on chip resources
like scratchpads or registers. Experimental results on six benchmarks and four
industry scale practical models are encouraging. Overall,
\emph{FusionStitching} can reach up to 5.7x speedup compared to Tensorflow
baseline, and achieves 1.25x to 1.85x performance speedups compared to current
state of the art, with 1.4x on average (geometric mean).
