  Our Visual Analytics (VA) tool ScrutinAI supports human analysts to
investigate interactively model performanceand data sets. Model performance
depends on labeling quality to a large extent. In particular in medical
settings, generation of high quality labels requires in depth expert knowledge
and is very costly. Often, data sets are labeled by collecting opinions of
groups of experts. We use our VA tool to analyse the influence of label
variations between different experts on the model performance. ScrutinAI
facilitates to perform a root cause analysis that distinguishes weaknesses of
deep neural network (DNN) models caused by varying or missing labeling quality
from true weaknesses. We scrutinize the overall detection of intracranial
hemorrhages and the more subtle differentiation between subtypes in a publicly
available data set.
