  People increasingly search online for answers to their medical questions but
the rate at which medical questions are asked online significantly exceeds the
capacity of qualified people to answer them. This leaves many questions
unanswered or inadequately answered. Many of these questions are not unique,
and reliable identification of similar questions would enable more efficient
and effective question answering schema. COVID-19 has only exacerbated this
problem. Almost every government agency and healthcare organization has tried
to meet the informational need of users by building online FAQs, but there is
no way for people to ask their question and know if it is answered on one of
these pages. While many research efforts have focused on the problem of general
question similarity, these approaches do not generalize well to domains that
require expert knowledge to determine semantic similarity, such as the medical
domain. In this paper, we show how a double fine-tuning approach of pretraining
a neural network on medical question-answer pairs followed by fine-tuning on
medical question-question pairs is a particularly useful intermediate task for
the ultimate goal of determining medical question similarity. While other
pretraining tasks yield an accuracy below 78.7% on this task, our model
achieves an accuracy of 82.6% with the same number of training examples, an
accuracy of 80.0% with a much smaller training set, and an accuracy of 84.5%
when the full corpus of medical question-answer data is used. We also describe
a currently live system that uses the trained model to match user questions to
COVID-related FAQs.
