  An important issue impacting healthcare is a lack of available experts.
Machine learning (ML) models could resolve this by aiding in diagnosing
patients. However, creating datasets large enough to train these models is
expensive. We evaluated large language models (LLMs) for data creation. Using
Autism Spectrum Disorders (ASD), we prompted ChatGPT and GPT-Premium to
generate 4,200 synthetic observations to augment existing medical data. Our
goal is to label behaviors corresponding to autism criteria and improve model
accuracy with synthetic training data. We used a BERT classifier pre-trained on
biomedical literature to assess differences in performance between models. A
random sample (N=140) from the LLM-generated data was evaluated by a clinician
and found to contain 83% correct example-label pairs. Augmenting data increased
recall by 13% but decreased precision by 16%, correlating with higher quality
and lower accuracy across pairs. Future work will analyze how different
synthetic data traits affect ML outcomes.
