  The availability of large public datasets and the increased amount of
computing power have shifted the interest of the medical community to
high-performance algorithms. However, little attention is paid to the quality
of the data and their annotations. High performance on benchmark datasets may
be reported without considering possible shortcuts or artifacts in the data,
besides, models are not tested on subpopulation groups. With this work, we aim
to raise awareness about shortcuts problems. We validate previous findings, and
present a case study on chest X-rays using two publicly available datasets. We
share annotations for a subset of pneumothorax images with drains. We conclude
with general recommendations for medical image classification.
