  Autonomous driving systems (ADSs) are capable of sensing the environment and
making driving decisions autonomously. These systems are safety-critical, and
testing them is one of the important approaches to ensure their safety.
However, due to the inherent complexity of ADSs and the high dimensionality of
their operating environment, the number of possible test scenarios for ADSs is
infinite. Besides, the operating environment of ADSs is dynamic, continuously
evolving, and full of uncertainties, which requires a testing approach adaptive
to the environment. In addition, existing ADS testing techniques have limited
effectiveness in ensuring the realism of test scenarios, especially the realism
of weather conditions and their changes over time. Recently, reinforcement
learning (RL) has demonstrated great potential in addressing challenging
problems, especially those requiring constant adaptations to dynamic
environments. To this end, we present DeepQTest, a novel ADS testing approach
that uses RL to learn environment configurations with a high chance of
revealing abnormal ADS behaviors. Specifically, DeepQTest employs Deep
Q-Learning and adopts three safety and comfort measures to construct the reward
functions. To ensure the realism of generated scenarios, DeepQTest defines a
set of realistic constraints and introduces real-world weather conditions into
the simulated environment. We employed three comparison baselines, i.e.,
random, greedy, and a state-of-the-art RL-based approach DeepCOllision, for
evaluating DeepQTest on an industrial-scale ADS. Evaluation results show that
DeepQTest demonstrated significantly better effectiveness in terms of
generating scenarios leading to collisions and ensuring scenario realism
compared with the baselines. In addition, among the three reward functions
implemented in DeepQTest, Time-To-Collision is recommended as the best design
according to our study.
