  To generate evidence regarding the safety and efficacy of artificial
intelligence (AI) enabled medical devices, AI models need to be evaluated on a
diverse population of patient cases, some of which may not be readily
available. We propose an evaluation approach for testing medical imaging AI
models that relies on in silico imaging pipelines in which stochastic digital
models of human anatomy (in object space) with and without pathology are imaged
using a digital replica imaging acquisition system to generate realistic
synthetic image datasets. Here, we release M-SYNTH, a dataset of cohorts with
four breast fibroglandular density distributions imaged at different exposure
levels using Monte Carlo x-ray simulations with the publicly available Virtual
Imaging Clinical Trial for Regulatory Evaluation (VICTRE) toolkit. We utilize
the synthetic dataset to analyze AI model performance and find that model
performance decreases with increasing breast density and increases with higher
mass density, as expected. As exposure levels decrease, AI model performance
drops with the highest performance achieved at exposure levels lower than the
nominal recommended dose for the breast type.
