  The approaches by which the machine learning and clinical research
communities utilize real world data (RWD), including data captured in the
electronic health record (EHR), vary dramatically. While clinical researchers
cautiously use RWD for clinical investigations, ML for healthcare teams consume
public datasets with minimal scrutiny to develop new algorithms. This study
bridges this gap by developing and validating ML-DQA, a data quality assurance
framework grounded in RWD best practices. The ML-DQA framework is applied to
five ML projects across two geographies, different medical conditions, and
different cohorts. A total of 2,999 quality checks and 24 quality reports were
generated on RWD gathered on 247,536 patients across the five projects. Five
generalizable practices emerge: all projects used a similar method to group
redundant data element representations; all projects used automated utilities
to build diagnosis and medication data elements; all projects used a common
library of rules-based transformations; all projects used a unified approach to
assign data quality checks to data elements; and all projects used a similar
approach to clinical adjudication. An average of 5.8 individuals, including
clinicians, data scientists, and trainees, were involved in implementing ML-DQA
for each project and an average of 23.4 data elements per project were either
transformed or removed in response to ML-DQA. This study demonstrates the
importance role of ML-DQA in healthcare projects and provides teams a framework
to conduct these essential activities.
