  We evaluated how clinicians approach clinical decision-making when given
findings from both randomized controlled trials (RCTs) and machine learning
(ML) models. To do so, we designed a clinical decision support system (CDSS)
that displays survival curves and adverse event information from a synthetic
RCT and ML model for 12 patients with multiple myeloma. We conducted an
interventional study in a simulated setting to evaluate how clinicians
synthesized the available data to make treatment decisions. Participants were
invited to participate in a follow-up interview to discuss their choices in an
open-ended format. When ML model results were concordant with RCT results,
physicians had increased confidence in treatment choice compared to when they
were given RCT results alone. When ML model results were discordant with RCT
results, the majority of physicians followed the ML model recommendation in
their treatment selection. Perceived reliability of the ML model was
consistently higher after physicians were provided with data on how it was
trained and validated. Follow-up interviews revealed four major themes: (1)
variability in what variables participants used for decision-making, (2)
perceived advantages to an ML model over RCT data, (3) uncertainty around
decision-making when the ML model quality was poor, and (4) perception that
this type of study is an important thought exercise for clinicians. Overall,
ML-based CDSSs have the potential to change treatment decisions in cancer
management. However, meticulous development and validation of these systems as
well as clinician training are required before deployment.
