  Autonomous Driving System (ADS) testing is crucial in ADS development, with
the current primary focus being on safety. However, the evaluation of
non-safety-critical performance, particularly the ADS's ability to make optimal
decisions and produce optimal paths for autonomous vehicles (AVs), is equally
vital to ensure the intelligence and reduce risks of AVs. Currently, there is
little work dedicated to assessing ADSs' optimal decision-making performance
due to the lack of corresponding oracles and the difficulty in generating
scenarios with non-optimal decisions. In this paper, we focus on evaluating the
decision-making quality of an ADS and propose the first method for detecting
non-optimal decision scenarios (NoDSs), where the ADS does not compute optimal
paths for AVs. Firstly, to deal with the oracle problem, we propose a novel
metamorphic relation (MR) aimed at exposing violations of optimal decisions.
The MR identifies the property that the ADS should retain optimal decisions
when the optimal path remains unaffected by non-invasive changes. Subsequently,
we develop a new framework, Decictor, designed to generate NoDSs efficiently.
Decictor comprises three main components: Non-invasive Mutation, MR Check, and
Feedback. The Non-invasive Mutation ensures that the original optimal path in
the mutated scenarios is not affected, while the MR Check is responsible for
determining whether non-optimal decisions are made. To enhance the
effectiveness of identifying NoDSs, we design a feedback metric that combines
both spatial and temporal aspects of the AV's movement. We evaluate Decictor on
Baidu Apollo, an open-source and production-grade ADS. The experimental results
validate the effectiveness of Decictor in detecting non-optimal decisions of
ADSs. Our work provides valuable and original insights into evaluating the
non-safety-critical performance of ADSs.
