  The increasing levels of software- and data-intensive driving automation call
for an evolution of automotive software testing. As a recommended practice of
the Verification and Validation (V&V) process of ISO/PAS 21448, a candidate
standard for safety of the intended functionality for road vehicles,
simulation-based testing has the potential to reduce both risks and costs.
There is a growing body of research on devising test automation techniques
using simulators for Advanced Driver-Assistance Systems (ADAS). However, how
similar are the results if the same test scenarios are executed in different
simulators? We conduct a replication study of applying a Search-Based Software
Testing (SBST) solution to a real-world ADAS (PeVi, a pedestrian vision
detection system) using two different commercial simulators, namely,
TASS/Siemens PreScan and ESI Pro-SiVIC. Based on a minimalistic scene, we
compare critical test scenarios generated using our SBST solution in these two
simulators. We show that SBST can be used to effectively and efficiently
generate critical test scenarios in both simulators, and the test results
obtained from the two simulators can reveal several weaknesses of the ADAS
under test. However, executing the same test scenarios in the two simulators
leads to notable differences in the details of the test outputs, in particular,
related to (1) safety violations revealed by tests, and (2) dynamics of cars
and pedestrians. Based on our findings, we recommend future V&V plans to
include multiple simulators to support robust simulation-based testing and to
base test objectives on measures that are less dependant on the internals of
the simulators.
