  The rapidly-changing deep learning landscape presents a unique opportunity
for building inference accelerators optimized for specific datacenter-scale
workloads. We propose Full-stack Accelerator Search Technique (FAST), a
hardware accelerator search framework that defines a broad optimization
environment covering key design decisions within the hardware-software stack,
including hardware datapath, software scheduling, and compiler passes such as
operation fusion and tensor padding. In this paper, we analyze bottlenecks in
state-of-the-art vision and natural language processing (NLP) models, including
EfficientNet and BERT, and use FAST to design accelerators capable of
addressing these bottlenecks. FAST-generated accelerators optimized for single
workloads improve Perf/TDP by 3.7x on average across all benchmarks compared to
TPU-v3. A FAST-generated accelerator optimized for serving a suite of workloads
improves Perf/TDP by 2.4x on average compared to TPU-v3. Our return on
investment analysis shows that FAST-generated accelerators can potentially be
practical for moderate-sized datacenter deployments.
