  The onset of the COVID-19 pandemic accentuated the need for access to
biomedical literature to answer timely and disease-specific questions. During
the early days of the pandemic, one of the biggest challenges we faced was the
lack of peer-reviewed biomedical articles on COVID-19 that could be used to
train machine learning models for question answering (QA). In this paper, we
explore the roles weak supervision and data augmentation play in training deep
neural network QA models. First, we investigate whether labels generated
automatically from the structured abstracts of scholarly papers using an
information retrieval algorithm, BM25, provide a weak supervision signal to
train an extractive QA model. We also curate new QA pairs using information
retrieval techniques, guided by the clinicaltrials.gov schema and the
structured abstracts of articles, in the absence of annotated data from
biomedical domain experts. Furthermore, we explore augmenting the training data
of a deep neural network model with linguistic features from external sources
such as lexical databases to account for variations in word morphology and
meaning. To better utilize our training data, we apply curriculum learning to
domain adaptation, fine-tuning our QA model in stages based on characteristics
of the QA pairs. We evaluate our methods in the context of QA models at the
core of a system to answer questions about COVID-19.
