  As one of the emerging challenges in Automated Machine Learning, the
Hardware-aware Neural Architecture Search (HW-NAS) tasks can be treated as
black-box multi-objective optimization problems (MOPs). An important
application of HW-NAS is real-time semantic segmentation, which plays a pivotal
role in autonomous driving scenarios. The HW-NAS for real-time semantic
segmentation inherently needs to balance multiple optimization objectives,
including model accuracy, inference speed, and hardware-specific
considerations. Despite its importance, benchmarks have yet to be developed to
frame such a challenging task as multi-objective optimization. To bridge the
gap, we introduce a tailored streamline to transform the task of HW-NAS for
real-time semantic segmentation into standard MOPs. Building upon the
streamline, we present a benchmark test suite, CitySeg/MOP, comprising fifteen
MOPs derived from the Cityscapes dataset. The CitySeg/MOP test suite is
integrated into the EvoXBench platform to provide seamless interfaces with
various programming languages (e.g., Python and MATLAB) for instant fitness
evaluations. We comprehensively assessed the CitySeg/MOP test suite on various
multi-objective evolutionary algorithms, showcasing its versatility and
practicality. Source codes are available at
https://github.com/EMI-Group/evoxbench.
