  Image quality assessment (IQA) is not just indispensable in clinical practice
to ensure high standards, but also in the development stage of novel algorithms
that operate on medical images with reference data. This paper provides a
structured and comprehensive collection of examples where the two most common
full reference (FR) image quality measures prove to be unsuitable for the
assessment of novel algorithms using different kinds of medical images,
including real-world MRI, CT, OCT, X-Ray, digital pathology and photoacoustic
imaging data. In particular, the FR-IQA measures PSNR and SSIM are known and
tested for working successfully in many natural imaging tasks, but
discrepancies in medical scenarios have been noted in the literature.
Inconsistencies arising in medical images are not surprising, as they have very
different properties than natural images which have not been targeted nor
tested in the development of the mentioned measures, and therefore might imply
wrong judgement of novel methods for medical images. Therefore, improvement is
urgently needed in particular in this era of AI to increase explainability,
reproducibility and generalizability in machine learning for medical imaging
and beyond. On top of the pitfalls we will provide ideas for future research as
well as suggesting guidelines for the usage of FR-IQA measures applied to
medical images.
