  How can we study social interactions on evolving topics at a mass scale? Over
the past decade, researchers from diverse fields such as economics, political
science, and public health have often done this by querying Twitter's public
API endpoints with hand-picked topical keywords to search or stream
discussions. However, despite the API's accessibility, it remains difficult to
select and update keywords to collect high-quality data relevant to topics of
interest. In this paper, we propose an active learning method for rapidly
refining query keywords to increase both the yielded topic relevance and
dataset size. We leverage a large open-source COVID-19 Twitter dataset to
illustrate the applicability of our method in tracking Tweets around the key
sub-topics of Vaccine, Mask, and Lockdown. Our experiments show that our method
achieves an average topic-related keyword recall 2x higher than baselines. We
open-source our code along with a web interface for keyword selection to make
data collection from Twitter more systematic for researchers.
