  AIOps algorithms play a crucial role in the maintenance of microservice
systems. Many previous benchmarks' performance leaderboard provides valuable
guidance for selecting appropriate algorithms. However, existing AIOps
benchmarks mainly utilize offline datasets to evaluate algorithms. They cannot
consistently evaluate the performance of algorithms using real-time datasets,
and the operation scenarios for evaluation are static, which is insufficient
for effective algorithm selection. To address these issues, we propose an
evaluation-consistent and scenario-oriented evaluation framework named
MicroServo. The core idea is to build a live microservice benchmark to generate
real-time datasets and consistently simulate the specific operation scenarios
on it. MicroServo supports different leaderboards by selecting specific
algorithms and datasets according to the operation scenarios. It also supports
the deployment of various types of algorithms, enabling algorithms
hot-plugging. At last, we test MicroServo with three typical microservice
operation scenarios to demonstrate its efficiency and usability.
