  Increasingly large electronic health records (EHRs) provide an opportunity to
algorithmically learn medical knowledge. In one prominent example, a causal
health knowledge graph could learn relationships between diseases and symptoms
and then serve as a diagnostic tool to be refined with additional clinical
input. Prior research has demonstrated the ability to construct such a graph
from over 270,000 emergency department patient visits. In this work, we
describe methods to evaluate a health knowledge graph for robustness. Moving
beyond precision and recall, we analyze for which diseases and for which
patients the graph is most accurate. We identify sample size and unmeasured
confounders as major sources of error in the health knowledge graph. We
introduce a method to leverage non-linear functions in building the causal
graph to better understand existing model assumptions. Finally, to assess model
generalizability, we extend to a larger set of complete patient visits within a
hospital system. We conclude with a discussion on how to robustly extract
medical knowledge from EHRs.
