  Transparency in Machine Learning (ML), attempts to reveal the working
mechanisms of complex models. Transparent ML promises to advance human factors
engineering goals of human-centered AI in the target users. From a
human-centered design perspective, transparency is not a property of the ML
model but an affordance, i.e. a relationship between algorithm and user; as a
result, iterative prototyping and evaluation with users is critical to
attaining adequate solutions that afford transparency. However, following
human-centered design principles in healthcare and medical image analysis is
challenging due to the limited availability of and access to end users. To
investigate the state of transparent ML in medical image analysis, we conducted
a systematic review of the literature. Our review reveals multiple severe
shortcomings in the design and validation of transparent ML for medical image
analysis applications. We find that most studies to date approach transparency
as a property of the model itself, similar to task performance, without
considering end users during neither development nor evaluation. Additionally,
the lack of user research, and the sporadic validation of transparency claims
put contemporary research on transparent ML for medical image analysis at risk
of being incomprehensible to users, and thus, clinically irrelevant. To
alleviate these shortcomings in forthcoming research while acknowledging the
challenges of human-centered design in healthcare, we introduce the INTRPRT
guideline, a systematic design directive for transparent ML systems in medical
image analysis. The INTRPRT guideline suggests formative user research as the
first step of transparent model design to understand user needs and domain
requirements. Following this process produces evidence to support design
choices, and ultimately, increases the likelihood that the algorithms afford
transparency.
