  In medicine, machine learning (ML) datasets are often built using the
International Classification of Diseases (ICD) codes. As new models are being
developed, there is a need for larger datasets. However, ICD codes are intended
for billing. We aim to determine how suitable ICD codes are for creating
datasets to train ML models. We focused on a rare and common disease using the
All of Us database. First, we compared the patient cohort created using ICD
codes for Valley fever (coccidioidomycosis, CM) with that identified via
serological confirmation. Second, we compared two similarly created patient
cohorts for myocardial infarction (MI) patients. We identified significant
discrepancies between these two groups, and the patient overlap was small. The
CM cohort had 811 patients in the ICD-10 group, 619 patients in the
positive-serology group, and 24 with both. The MI cohort had 14,875 patients in
the ICD-10 group, 23,598 in the MI laboratory-confirmed group, and 6,531 in
both. Demographics, rates of disease symptoms, and other clinical data varied
across our case study cohorts.
