  The increasing attention on deep learning has tremendously spurred the design
of intelligence processing hardware. The variety of emerging intelligence
processors requires standard benchmarks for fair comparison and system
optimization (in both software and hardware). However, existing benchmarks are
unsuitable for benchmarking intelligence processors due to their non-diversity
and nonrepresentativeness. Also, the lack of a standard benchmarking
methodology further exacerbates this problem. In this paper, we propose
BENCHIP, a benchmark suite and benchmarking methodology for intelligence
processors. The benchmark suite in BENCHIP consists of two sets of benchmarks:
microbenchmarks and macrobenchmarks. The microbenchmarks consist of
single-layer networks. They are mainly designed for bottleneck analysis and
system optimization. The macrobenchmarks contain state-of-the-art industrial
networks, so as to offer a realistic comparison of different platforms. We also
propose a standard benchmarking methodology built upon an industrial software
stack and evaluation metrics that comprehensively reflect the various
characteristics of the evaluated intelligence processors. BENCHIP is utilized
for evaluating various hardware platforms, including CPUs, GPUs, and
accelerators. BENCHIP will be open-sourced soon.
