  Supervised machine learning utilizes large datasets, often with ground truth
labels annotated by humans. While some data points are easy to classify, others
are hard to classify, which reduces the inter-annotator agreement. This causes
noise for the classifier and might affect the user's perception of the
classifier's performance. In our research, we investigated whether the
classification difficulty of a data point influences how strongly a prediction
mistake reduces the "perceived accuracy". In an experimental online study, 225
participants interacted with three fictive classifiers with equal accuracy
(73%). The classifiers made prediction mistakes on three different types of
data points (easy, difficult, impossible). After the interaction, participants
judged the classifier's accuracy. We found that not all prediction mistakes
reduced the perceived accuracy equally. Furthermore, the perceived accuracy
differed significantly from the calculated accuracy. To conclude, accuracy and
related measures seem unsuitable to represent how users perceive the
performance of classifiers.
