  This report represents a roadmap for integrating Artificial Intelligence
(AI)-based image analysis algorithms into existing Radiology workflows such
that: (1) radiologists can significantly benefit from enhanced automation in
various imaging tasks due to AI; and (2) radiologists' feedback is utilized to
further improve the AI application. This is achieved by establishing three
maturity levels where: (1) research enables the visualization of AI-based
results/annotations by radiologists without generating new patient records; (2)
production allows the AI-based system to generate results stored in an
institution's Picture Archiving and Communication System; and (3) feedback
equips radiologists with tools for editing the AI inference results for
periodic retraining of the deployed AI systems, thereby allowing the continuous
organic improvement of AI-based radiology-workflow solutions. A case study
(i.e., detection of brain metastases with T1-weighted contrast-enhanced 3D MRI)
illustrates the deployment details of a particular AI-based application
according to the aforementioned maturity levels. It is shown that the given AI
application significantly improves with the feedback coming from radiologists;
the number of incorrectly detected brain metastases (false positives) reduces
from 14.2 to 9.12 per patient with the number of subsequently annotated
datasets increasing from 93 to 217 as a result of radiologist adjudication.
