  Background. Large Language Models (LLMs) hold promise for improving genetic
variant literature review in clinical testing. We assessed Generative
Pretrained Transformer 4's (GPT-4) performance, nondeterminism, and drift to
inform its suitability for use in complex clinical processes. Methods. A
2-prompt process for classification of functional evidence was optimized using
a development set of 45 articles. The prompts asked GPT-4 to supply all
functional data present in an article related to a variant or indicate that no
functional evidence is present. For articles indicated as containing functional
evidence, a second prompt asked GPT-4 to classify the evidence into pathogenic,
benign, or intermediate/inconclusive categories. A final test set of 72
manually classified articles was used to test performance. Results. Over a
2.5-month period (Dec 2023-Feb 2024), we observed substantial differences in
intraday (nondeterminism) and across day (drift) results, which lessened after
1/18/24. This variability is seen within and across models in the GPT-4 series,
affecting different performance statistics to different degrees. Twenty runs
after 1/18/24 identified articles containing functional evidence with 92.2%
sensitivity, 95.6% positive predictive value (PPV) and 86.3% negative
predictive value (NPV). The second prompt's identified pathogenic functional
evidence with 90.0% sensitivity, 74.0% PPV and 95.3% NVP and for benign
evidence with 88.0% sensitivity, 76.6% PPV and 96.9% NVP. Conclusion.
Nondeterminism and drift within LLMs must be assessed and monitored when
introducing LLM based functionality into clinical workflows. Failing to do this
assessment or accounting for these challenges could lead to incorrect or
missing information that is critical for patient care. The performance of our
prompts appears adequate to assist in article prioritization but not in
automated decision making.
