  Deep learning models for organ contouring in radiotherapy are poised for
clinical usage, but currently, there exist few tools for automated quality
assessment (QA) of the predicted contours. Using Bayesian models and their
associated uncertainty, one can potentially automate the process of detecting
inaccurate predictions. We investigate two Bayesian models for auto-contouring,
DropOut and FlipOut, using a quantitative measure - expected calibration error
(ECE) and a qualitative measure - region-based accuracy-vs-uncertainty (R-AvU)
graphs. It is well understood that a model should have low ECE to be considered
trustworthy. However, in a QA context, a model should also have high
uncertainty in inaccurate regions and low uncertainty in accurate regions. Such
behaviour could direct visual attention of expert users to potentially
inaccurate regions, leading to a speed up in the QA process. Using R-AvU
graphs, we qualitatively compare the behaviour of different models in accurate
and inaccurate regions. Experiments are conducted on the MICCAI2015 Head and
Neck Segmentation Challenge and on the DeepMindTCIA CT dataset using three
models: DropOut-DICE, Dropout-CE (Cross Entropy) and FlipOut-CE. Quantitative
results show that DropOut-DICE has the highest ECE, while Dropout-CE and
FlipOut-CE have the lowest ECE. To better understand the difference between
DropOut-CE and FlipOut-CE, we use the R-AvU graph which shows that FlipOut-CE
has better uncertainty coverage in inaccurate regions than DropOut-CE. Such a
combination of quantitative and qualitative metrics explores a new approach
that helps to select which model can be deployed as a QA tool in clinical
settings.
