  Building multi-domain AI agents is a challenging task and an open problem in
the area of AI. Within the domain of dialog, the ability to orchestrate
multiple independently trained dialog agents, or skills, to create a unified
system is of particular significance. In this work, we study the task of online
posterior dialog orchestration, where we define posterior orchestration as the
task of selecting a subset of skills which most appropriately answer a user
input using features extracted from both the user input and the individual
skills. To account for the various costs associated with extracting skill
features, we consider online posterior orchestration under a skill execution
budget. We formalize this setting as Context Attentive Bandit with Observations
(CABO), a variant of context attentive bandits, and evaluate it on simulated
non-conversational and proprietary conversational datasets.
