  Software metrics such as coverage and mutation scores have been extensively
explored for the automated quality assessment of test suites. While traditional
tools rely on such quantifiable software metrics, the field of self-driving
cars (SDCs) has primarily focused on simulation-based test case generation
using quality metrics such as the out-of-bound (OOB) parameter to determine if
a test case fails or passes. However, it remains unclear to what extent this
quality metric aligns with the human perception of the safety and realism of
SDCs, which are critical aspects in assessing SDC behavior. To address this
gap, we conducted an empirical study involving 50 participants to investigate
the factors that determine how humans perceive SDC test cases as safe, unsafe,
realistic, or unrealistic. To this aim, we developed a framework leveraging
virtual reality (VR) technologies, called SDC-Alabaster, to immerse the study
participants into the virtual environment of SDC simulators. Our findings
indicate that the human assessment of the safety and realism of failing and
passing test cases can vary based on different factors, such as the test's
complexity and the possibility of interacting with the SDC. Especially for the
assessment of realism, the participants' age as a confounding factor leads to a
different perception. This study highlights the need for more research on SDC
simulation testing quality metrics and the importance of human perception in
evaluating SDC behavior.
