  AI is promising in assisting UX evaluators with analyzing usability tests,
but its judgments are typically presented as non-interactive visualizations.
Evaluators may have questions about test recordings, but have no way of asking
them. Interactive conversational assistants provide a Q&A dynamic that may
improve analysis efficiency and evaluator autonomy. To understand the full
range of analysis-related questions, we conducted a Wizard-of-Oz design probe
study with 20 participants who interacted with simulated AI assistants via text
or voice. We found that participants asked for five categories of information:
user actions, user mental model, help from the AI assistant, product and task
information, and user demographics. Those who used the text assistant asked
more questions, but the question lengths were similar. The text assistant was
perceived as significantly more efficient, but both were rated equally in
satisfaction and trust. We also provide design considerations for future
conversational AI assistants for UX evaluation.
