  Wizard of OZ (WOZ) is a well-established method for simulating the
functionality and user experience of future systems. Using a human wizard to
mimic certain operations of a potential system is particularly useful in
situations where extensive engineering effort would otherwise be needed to
explore the design possibilities offered by such operations. The WOZ method has
been widely used in connection with speech and language technologies, but
advances in sensor technology and pattern recognition as well as new
application areas such as human-robot interaction have made it increasingly
relevant to the design of a wider range of interactive systems. In such cases
achieving acceptable performance at the user interface level often hinges on
resource intensive improvements such as domain tuning, which are better done
once the overall design is relatively stable. While WOZ is recognised as a
valuable prototyping technique, surprisingly little effort has been put into
exploring it from a methodological point of view. Starting from a survey of the
literature, this paper presents a systematic investigation and analysis of the
design space for WOZ for language technology applications, and proposes a
generic architecture for tool support that supports the integration of
components for speech recognition and synthesis as well as for machine
translation. This architecture is instantiated in WebWOZ - a new web-based
open-source WOZ prototyping platform. The viability of generic support is
explored empirically through a series of evaluations. Researchers from a
variety of backgrounds were able to create experiments, independent of their
previous experience with WOZ. The approach was further validated through a
number of real experiments, which also helped to identify a number of
possibilities for additional support, and flagged potential issues relating to
consistency in Wizard performance.
