  Effective optimization is essential for real-world interactive systems to
provide a satisfactory user experience in response to changing user behavior.
However, it is often challenging to find an objective to optimize for
interactive systems (e.g., policy learning in task-oriented dialog systems).
Generally, such objectives are manually crafted and rarely capture complex user
needs in an accurate manner. We propose an approach that infers the objective
directly from observed user interactions. These inferences can be made
regardless of prior knowledge and across different types of user behavior. We
introduce Interactive System Optimizer (ISO), a novel algorithm that uses these
inferred objectives for optimization. Our main contribution is a new general
principled approach to optimizing interactive systems using data-driven
objectives. We demonstrate the high effectiveness of ISO over several
simulations.
