  Business-specific Frequently Asked Questions (FAQ) retrieval in task-oriented
dialog systems poses unique challenges vis-\`a-vis community based FAQs. Each
FAQ question represents an intent which is usually an umbrella term for many
related user queries. We evaluate performance for such Business FAQs both with
standard FAQ retrieval techniques using query-Question (q-Q) similarity and
few-shot intent detection techniques. Implementing a real world solution for
FAQ retrieval in order to support multiple tenants (FAQ sets) entails
optimizing speed, accuracy and cost. We propose a novel approach to scale
multi-tenant FAQ applications in real-world context by contrastive fine-tuning
of the last layer in sentence Bi-Encoders along with tenant-specific weight
switching.
