  Intelligent personal assistants (IPA) enable voice applications that
facilitate people's daily tasks. However, due to the complexity and ambiguity
of voice requests, some requests may not be handled properly by the standard
natural language understanding (NLU) component. In such cases, a simple reply
like "Sorry, I don't know" hurts the user's experience and limits the
functionality of IPA. In this paper, we propose a two-stage
shortlister-reranker recommender system to match third-party voice applications
(skills) to unhandled utterances. In this approach, a skill shortlister is
proposed to retrieve candidate skills from the skill catalog by calculating
both lexical and semantic similarity between skills and user requests. We also
illustrate how to build a new system by using observed data collected from a
baseline rule-based system, and how the exposure biases can generate
discrepancy between offline and human metrics. Lastly, we present two
relabeling methods that can handle the incomplete ground truth, and mitigate
exposure bias. We demonstrate the effectiveness of our proposed system through
extensive offline experiments. Furthermore, we present online A/B testing
results that show a significant boost on user experience satisfaction.
