  Safety is one of the main challenges that prohibit autonomous vehicles (AV),
requiring them to be well tested ahead of being allowed on the road. In
comparison with road tests, simulators allow us to validate the AV conveniently
and affordably. However, it remains unclear how to best use the AV-based
simulator system for testing effectively. Our paper presents an empirical
testing of AV simulator system that combines the SVL simulator and the Apollo
platform. We propose 576 test cases which are inspired by four naturalistic
driving situations with pedestrians and surrounding cars. We found that the SVL
can imitate realistic safe and collision situations; and at the same time,
Apollo can drive the car quite safely. On the other hand, we noted that the
system failed to detect pedestrians or vehicles on the road in three out of
four classes, accounting for 10.0% total number of scenarios tested. We further
applied metamorphic testing to identify inconsistencies in the system with
additional 486 test cases. We then discussed some insights into the scenarios
that may cause hazardous situations in real life. In summary, this paper
provides a new empirical evidence to strengthen the assertion that the
simulator-based system can be an indispensable tool for a comprehensive testing
of the AV.
