  There is a growing demand for shifting the delivery of AI capability from
data centers on the cloud to edge or end devices, exemplified by the fast
emerging real-time AI-based apps running on smartphones, AR/VR devices,
autonomous vehicles, and various IoT devices. The shift has however been
seriously hampered by the large growing gap between DNN computing demands and
the computing power on edge or end devices. This article presents the design of
XGen, an optimizing framework for DNN designed to bridge the gap. XGen takes
cross-cutting co-design as its first-order consideration. Its full-stack
AI-oriented optimizations consist of a number of innovative optimizations at
every layer of the DNN software stack, all designed in a cooperative manner.
The unique technology makes XGen able to optimize various DNNs, including those
with an extreme depth (e.g., BERT, GPT, other transformers), and generate code
that runs several times faster than those from existing DNN frameworks, while
delivering the same level of accuracy.
