  Benchmarking involves designing scientific test methods, tools, and
frameworks to quantitatively and comparably assess specific performance
indicators of certain test subjects. With the development of artificial
intelligence, AI benchmarking datasets such as ImageNet and DataPerf have
gradually become consensus standards in both academic and industrial fields.
However, constructing a benchmarking framework remains a significant challenge
in the open-source domain due to the diverse range of data types, the wide
array of research issues, and the intricate nature of collaboration networks.
This paper introduces OpenPerf, a benchmarking framework designed for the
sustainable development of the open-source ecosystem. This framework defines 9
task benchmarking tasks in the open-source research, encompassing 3 data types:
time series, text, and graphics, and addresses 6 research problems including
regression, classification, recommendation, ranking, network building, and
anomaly detection. Based on the above tasks, we implemented 3 data science task
benchmarks, 2 index-based benchmarks, and 1 standard benchmark. Notably, the
index-based benchmarks have been adopted by the China Electronics
Standardization Institute as evaluation criteria for open-source community
governance. Additionally, we have developed a comprehensive toolkit for
OpenPerf, which not only offers robust data management, tool integration, and
user interface capabilities but also adopts a Benchmarking-as-a-Service (BaaS)
model to serve academic institutions, industries, and foundations. Through its
application in renowned companies and institutions such as Alibaba, Ant Group,
and East China Normal University, we have validated OpenPerf's pivotal role in
the healthy evolution of the open-source ecosystem.
