  Image quality assessment (IQA) is standard practice in the development stage
of novel machine learning algorithms that operate on images. The most commonly
used IQA measures have been developed and tested for natural images, but not in
the medical setting. Reported inconsistencies arising in medical images are not
surprising, as they have different properties than natural images. In this
study, we test the applicability of common IQA measures for medical image data
by comparing their assessment to manually rated chest X-ray (5 experts) and
photoacoustic image data (1 expert). Moreover, we include supplementary studies
on grayscale natural images and accelerated brain MRI data. The results of all
experiments show a similar outcome in line with previous findings for medical
imaging: PSNR and SSIM in the default setting are in the lower range of the
result list and HaarPSI outperforms the other tested measures in the overall
performance. Also among the top performers in our medical experiments are the
full reference measures DISTS, FSIM, LPIPS and MS-SSIM. Generally, the results
on natural images yield considerably higher correlations, suggesting that the
additional employment of tailored IQA measures for medical imaging algorithms
is needed.
