  This work-in-progress paper describes a vision, i.e., that of fast and
reliable software user experience studies conducted with the help from the
crowd. Commonly, user studies are controlled in-lab activities that require the
instruction, monitoring, interviewing and compensation of a number of
participants that are typically hard to recruit. The goal of this work is to
study which user study methods can instead be crowdsourced to generic audiences
to enable the conduct of user studies without the need for expensive lab
experiments. The challenge is understanding how to conduct crowdsourced studies
without giving up too many of the guarantees in-lab settings are able to
provide.
