  Modern real-world application scenarios like Internet services consist of a
diversity of AI and non-AI modules with huge code sizes and long and
complicated execution paths, which raises serious benchmarking or evaluating
challenges. Using AI components or micro benchmarks alone can lead to
error-prone conclusions. This paper presents a methodology to attack the above
challenge. We formalize a real-world application scenario as a Directed Acyclic
Graph-based model and propose the rules to distill it into a permutation of
essential AI and non-AI tasks, which we call a scenario benchmark. Together
with seventeen industry partners, we extract nine typical scenario benchmarks.
We design and implement an extensible, configurable, and flexible benchmark
framework. We implement two Internet service AI scenario benchmarks based on
the framework as proxies to two real-world application scenarios. We consider
scenario, component, and micro benchmarks as three indispensable parts for
evaluating. Our evaluation shows the advantage of our methodology against using
component or micro AI benchmarks alone. The specifications, source code,
testbed, and results are publicly available from
\url{https://www.benchcouncil.org/aibench/scenario/}.
