  Documentation burden is a major contributor to clinician burnout, which is
rising nationally and is an urgent threat to our ability to care for patients.
Artificial intelligence (AI) chatbots, such as ChatGPT, could reduce clinician
burden by assisting with documentation. Although many hospitals are actively
integrating such systems into electronic medical record systems, AI chatbots
utility and impact on clinical decision-making have not been studied for this
intended use. We are the first to examine the utility of large language models
in assisting clinicians draft responses to patient questions. In our two-stage
cross-sectional study, 6 oncologists responded to 100 realistic synthetic
cancer patient scenarios and portal messages developed to reflect common
medical situations, first manually, then with AI assistance.
  We find AI-assisted responses were longer, less readable, but provided
acceptable drafts without edits 58% of time. AI assistance improved efficiency
77% of time, with low harm risk (82% safe). However, 7.7% unedited AI responses
could severely harm. In 31% cases, physicians thought AI drafts were
human-written. AI assistance led to more patient education recommendations,
fewer clinical actions than manual responses. Results show promise for AI to
improve clinician efficiency and patient care through assisting documentation,
if used judiciously. Monitoring model outputs and human-AI interaction remains
crucial for safe implementation.
