  In a recent study, Reinforcement Learning (RL) used in combination with
many-objective search, has been shown to outperform alternative techniques
(random search and many-objective search) for online testing of Deep Neural
Network-enabled systems. The empirical evaluation of these techniques was
conducted on a state-of-the-art Autonomous Driving System (ADS). This work is a
replication and extension of that empirical study. Our replication shows that
RL does not outperform pure random test generation in a comparison conducted
under the same settings of the original study, but with no confounding factor
coming from the way collisions are measured. Our extension aims at eliminating
some of the possible reasons for the poor performance of RL observed in our
replication: (1) the presence of reward components providing contrasting or
useless feedback to the RL agent; (2) the usage of an RL algorithm (Q-learning)
which requires discretization of an intrinsically continuous state space.
Results show that our new RL agent is able to converge to an effective policy
that outperforms random testing. Results also highlight other possible
improvements, which open to further investigations on how to best leverage RL
for online ADS testing.
