  Objective: Social media-based public health research is crucial for epidemic
surveillance, but most studies identify relevant corpora with keyword-matching.
This study develops a system to streamline the process of curating colloquial
medical dictionaries. We demonstrate the pipeline by curating a UMLS-colloquial
symptom dictionary from COVID-19-related tweets as proof of concept. Methods:
COVID-19-related tweets from February 1, 2020, to April 30, 2022 were used. The
pipeline includes three modules: a named entity recognition module to detect
symptoms in tweets; an entity normalization module to aggregate detected
entities; and a mapping module that iteratively maps entities to Unified
Medical Language System concepts. A random 500 entity sample were drawn from
the final dictionary for accuracy validation. Additionally, we conducted a
symptom frequency distribution analysis to compare our dictionary to a
pre-defined lexicon from previous research. Results: We identified 498,480
unique symptom entity expressions from the tweets. Pre-processing reduces the
number to 18,226. The final dictionary contains 38,175 unique expressions of
symptoms that can be mapped to 966 UMLS concepts (accuracy = 95%). Symptom
distribution analysis found that our dictionary detects more symptoms and is
effective at identifying psychiatric disorders like anxiety and depression,
often missed by pre-defined lexicons. Conclusions: This study advances public
health research by implementing a novel, systematic pipeline for curating
symptom lexicons from social media data. The final lexicon's high accuracy,
validated by medical professionals, underscores the potential of this
methodology to reliably interpret and categorize vast amounts of unstructured
social media data into actionable medical insights across diverse linguistic
and regional landscapes.
