  With the devastating outbreak of COVID-19, vaccines are one of the crucial
lines of defense against mass infection in this global pandemic. Given the
protection they provide, vaccines are becoming mandatory in certain social and
professional settings. This paper presents a classification model for detecting
COVID-19 vaccination related search queries, a machine learning model that is
used to generate search insights for COVID-19 vaccinations. The proposed method
combines and leverages advancements from modern state-of-the-art (SOTA) natural
language understanding (NLU) techniques such as pretrained Transformers with
traditional dense features. We propose a novel approach of considering dense
features as memory tokens that the model can attend to. We show that this new
modeling approach enables a significant improvement to the Vaccine Search
Insights (VSI) task, improving a strong well-established gradient-boosting
baseline by relative +15% improvement in F1 score and +14% in precision.
