  Large-scale pre-trained models have attracted extensive attention in the
research community and shown promising results on various tasks of natural
language processing. However, these pre-trained models are memory and
computation intensive, hindering their deployment into industrial online
systems like Ad Relevance. Meanwhile, how to design an effective yet efficient
model architecture is another challenging problem in online Ad Relevance.
Recently, AutoML shed new lights on architecture design, but how to integrate
it with pre-trained language models remains unsettled. In this paper, we
propose AutoADR (Automatic model design for AD Relevance) -- a novel end-to-end
framework to address this challenge, and share our experience to ship these
cutting-edge techniques into online Ad Relevance system at Microsoft Bing.
Specifically, AutoADR leverages a one-shot neural architecture search algorithm
to find a tailored network architecture for Ad Relevance. The search process is
simultaneously guided by knowledge distillation from a large pre-trained
teacher model (e.g. BERT), while taking the online serving constraints (e.g.
memory and latency) into consideration. We add the model designed by AutoADR as
a sub-model into the production Ad Relevance model. This additional sub-model
improves the Precision-Recall AUC (PR AUC) on top of the original Ad Relevance
model by 2.65X of the normalized shipping bar. More importantly, adding this
automatically designed sub-model leads to a statistically significant 4.6%
Bad-Ad ratio reduction in online A/B testing. This model has been shipped into
Microsoft Bing Ad Relevance Production model.
