  Today's Internet Services are undergoing fundamental changes and shifting to
an intelligent computing era where AI is widely employed to augment services.
In this context, many innovative AI algorithms, systems, and architectures are
proposed, and thus the importance of benchmarking and evaluating them rises.
However, modern Internet services adopt a microservice-based architecture and
consist of various modules. The diversity of these modules and complexity of
execution paths, the massive scale and complex hierarchy of datacenter
infrastructure, the confidential issues of data sets and workloads pose great
challenges to benchmarking. In this paper, we present the first
industry-standard Internet service AI benchmark suite---AIBench with seventeen
industry partners, including several top Internet service providers. AIBench
provides a highly extensible, configurable, and flexible benchmark framework
that contains loosely coupled modules. We identify sixteen prominent AI problem
domains like learning to rank, each of which forms an AI component benchmark,
from three most important Internet service domains: search engine, social
network, and e-commerce, which is by far the most comprehensive AI benchmarking
effort. On the basis of the AIBench framework, abstracting the real-world data
sets and workloads from one of the top e-commerce providers, we design and
implement the first end-to-end Internet service AI benchmark, which contains
the primary modules in the critical paths of an industry scale application and
is scalable to deploy on different cluster scales. The specifications, source
code, and performance numbers are publicly available from the benchmark council
web site http://www.benchcouncil.org/AIBench/index.html.
