  Objective: Text mining of clinical notes embedded in electronic medical
records is increasingly used to extract patient characteristics otherwise not
or only partly available, to assess their association with relevant health
outcomes. As manual data labeling needed to develop text mining models is
resource intensive, we investigated whether off-the-shelf text mining models
developed at external institutions, together with limited within-institution
labeled data, could be used to reliably extract study variables to conduct
association studies.
  Materials and Methods: We developed multiple text mining models on different
combinations of within-institution and external-institution data to extract
social factors from discharge reports of intensive care patients. Subsequently,
we assessed the associations between social factors and having a
do-not-resuscitate/intubate code. Results: Important differences were found
between associations based on manually labeled data compared to text-mined
social factors in three out of five cases. Adopting external-institution text
mining models using manually labeled within-institution data resulted in models
with higher F1-scores, but not in meaningfully different associations.
  Discussion: While text mining facilitated scaling analyses to larger samples
leading to discovering a larger number of associations, the estimates may be
unreliable. Confirmation is needed with better text mining models, ideally on a
larger manually labeled dataset.
  Conclusion: The currently used text mining models were not sufficiently
accurate to be used reliably in an association study. Model adaptation using
within-institution data did not improve the estimates. Further research is
needed to set conditions for reliable use of text mining in medical research.
