  Deep learning has enabled great strides in abdominal multi-organ
segmentation, even surpassing junior oncologists on common cases or organs.
However, robustness on corner cases and complex organs remains a challenging
open problem for clinical adoption. To investigate model robustness, we
collected and annotated the RAOS dataset comprising 413 CT scans ($\sim$80k 2D
images, $\sim$8k 3D organ annotations) from 413 patients each with 17 (female)
or 19 (male) labelled organs, manually delineated by oncologists. We grouped
scans based on clinical information into 1) diagnosis/radiotherapy (317
volumes), 2) partial excision without the whole organ missing (22 volumes), and
3) excision with the whole organ missing (74 volumes). RAOS provides a
potential benchmark for evaluating model robustness including organ
hallucination. It also includes some organs that can be very hard to access on
public datasets like the rectum, colon, intestine, prostate and seminal
vesicles. We benchmarked several state-of-the-art methods in these three
clinical groups to evaluate performance and robustness. We also assessed
cross-generalization between RAOS and three public datasets. This dataset and
comprehensive analysis establish a potential baseline for future robustness
research: \url{https://github.com/Luoxd1996/RAOS}.
