  The development of intrusion detection systems (IDS) for the in-vehicle
Controller Area Network (CAN) bus is one of the main efforts being taken to
secure the in-vehicle network against various cyberattacks, which have the
potential to cause vehicles to malfunction and result in dangerous accidents.
These CAN IDS are evaluated in disparate experimental conditions that vary in
terms of the workload used, the features used, the metrics reported, etc.,
which makes direct comparison difficult. Therefore, there have been several
benchmarking frameworks and comparative studies designed to evaluate CAN IDS in
similar experimental conditions to understand their relative performance and
facilitate the selection of the best CAN IDS for implementation in automotive
networks. This work provides a comprehensive survey of CAN IDS benchmarking
frameworks and comparative studies in the current literature. A CAN IDS
evaluation design space is also proposed in this work, which draws from the
wider CAN IDS literature. This is not only expected to serve as a guide for
designing CAN IDS evaluation experiments but is also used for categorizing
current benchmarking efforts. The surveyed works have been discussed on the
basis of the five aspects in the design space-namely IDS type, attack model,
evaluation type, workload generation, and evaluation metrics-and
recommendations for future work have been identified.
