  While we pay attention to the latest advances in clinical natural language
processing (NLP), we can notice some resistance in the clinical and
translational research community to adopt NLP models due to limited
transparency, interpretability, and usability. In this study, we proposed an
open natural language processing development framework. We evaluated it through
the implementation of NLP algorithms for the National COVID Cohort
Collaborative (N3C). Based on the interests in information extraction from
COVID-19 related clinical notes, our work includes 1) an open data annotation
process using COVID-19 signs and symptoms as the use case, 2) a
community-driven ruleset composing platform, and 3) a synthetic text data
generation workflow to generate texts for information extraction tasks without
involving human subjects. The corpora were derived from texts from three
different institutions (Mayo Clinic, University of Kentucky, University of
Minnesota). The gold standard annotations were tested with a single
institution's (Mayo) ruleset. This resulted in performances of 0.876, 0.706,
and 0.694 in F-scores for Mayo, Minnesota, and Kentucky test datasets,
respectively. The study as a consortium effort of the N3C NLP subgroup
demonstrates the feasibility of creating a federated NLP algorithm development
and benchmarking platform to enhance multi-institution clinical NLP study and
adoption. Although we use COVID-19 as a use case in this effort, our framework
is general enough to be applied to other domains of interest in clinical NLP.
