  This paper describes Waymo's Collision Avoidance Testing (CAT) methodology: a
scenario-based testing method that evaluates the safety of the Waymo Driver
Automated Driving Systems' (ADS) intended functionality in conflict situations
initiated by other road users that require urgent evasive maneuvers. Because
SAE Level 4 ADS are responsible for the dynamic driving task (DDT), when
engaged, without immediate human intervention, evaluating a Level 4 ADS using
scenario-based testing is difficult due to the potentially infinite number of
operational scenarios in which hazardous situations may unfold. To that end, in
this paper we first describe the safety test objectives for the CAT
methodology, including the collision and serious injury metrics and the
reference behavior model representing a non-impaired eyes on conflict human
driver used to form an acceptance criterion. Afterward, we introduce the
process for identifying potentially hazardous situations from a combination of
human data, ADS testing data, and expert knowledge about the product design and
associated Operational Design Domain (ODD). The test allocation and execution
strategy is presented next, which exclusively utilize simulations constructed
from sensor data collected on a test track, real-world driving, or from
simulated sensor data. The paper concludes with the presentation of results
from applying CAT to the fully autonomous ride-hailing service that Waymo
operates in San Francisco, California and Phoenix, Arizona. The iterative
nature of scenario identification, combined with over ten years of experience
of on-road testing, results in a scenario database that converges to a
representative set of responder role scenarios for a given ODD. Using Waymo's
virtual test platform, which is calibrated to data collected as part of many
years of ADS development, the CAT methodology provides a robust and scalable
safety evaluation.
