  The COVID-19 pandemic continues to bring up various topics discussed or
debated on social media. In order to explore the impact of pandemics on
people's lives, it is crucial to understand the public's concerns and attitudes
towards pandemic-related entities (e.g., drugs, vaccines) on social media.
However, models trained on existing named entity recognition (NER) or targeted
sentiment analysis (TSA) datasets have limited ability to understand
COVID-19-related social media texts because these datasets are not designed or
annotated from a medical perspective. This paper releases METS-CoV, a dataset
containing medical entities and targeted sentiments from COVID-19-related
tweets. METS-CoV contains 10,000 tweets with 7 types of entities, including 4
medical entity types (Disease, Drug, Symptom, and Vaccine) and 3 general entity
types (Person, Location, and Organization). To further investigate tweet users'
attitudes toward specific entities, 4 types of entities (Person, Organization,
Drug, and Vaccine) are selected and annotated with user sentiments, resulting
in a targeted sentiment dataset with 9,101 entities (in 5,278 tweets). To the
best of our knowledge, METS-CoV is the first dataset to collect medical
entities and corresponding sentiments of COVID-19-related tweets. We benchmark
the performance of classical machine learning models and state-of-the-art deep
learning models on NER and TSA tasks with extensive experiments. Results show
that the dataset has vast room for improvement for both NER and TSA tasks.
METS-CoV is an important resource for developing better medical social media
tools and facilitating computational social science research, especially in
epidemiology. Our data, annotation guidelines, benchmark models, and source
code are publicly available (https://github.com/YLab-Open/METS-CoV) to ensure
reproducibility.
