  Over the past few years, research has witnessed the advancement of deep
learning models trained on large datasets, some even encompassing millions of
examples. While these impressive performance on their hidden test sets, they
often underperform when assessed on external datasets. Recognizing the critical
role of generalization in medical AI development, many prestigious journals now
require reporting results both on the local hidden test set as well as on
external datasets before considering a study for publication. Effectively, the
field of medical AI has transitioned from the traditional usage of a single
dataset that is split into train and test to a more comprehensive framework
using multiple datasets, some of which are used for model development (source
domain) and others for testing (target domains). However, this new experimental
setting does not necessarily resolve the challenge of generalization. This is
because of the variability encountered in intended use and specificities across
hospital cultures making the idea of universally generalizable systems a myth.
On the other hand, the systematic, and a fortiori recurrent re-calibration, of
models at the individual hospital level, although ideal, may be overoptimistic
given the legal, regulatory and technical challenges that are involved.
Re-calibration using transfer learning may not even be possible in some
instances where reference labels of target domains are not available. In this
perspective we establish a hierarchical three-level scale system reflecting the
generalization level of a medical AI algorithm. This scale better reflects the
diversity of real-world medical scenarios per which target domain data for
re-calibration of models may or not be available and if it is, may or not have
reference labels systematically available.
