The first program a new computer programmer writes in any new programming language is the “Hello world!” program – a single line of code that prints “Hello world!” to the screen.
We ask, by analogy, what should be the first “program” a new user of crowdsourcing or human computation writes? “HelloCrowd!” is our answer.
Crowdsourcing and human computation are becoming ever more popular tools for answering questions, collecting data, and providing human judgment. At the same time, there is a disconnect between interest and ability, where potential new users of these powerful tools don’t know how to get started. Not everyone wants to take a graduate course in crowdsourcing just to get their first results. To fix this, we set out to build an interactive tutorial that could teach the fundamentals of crowdsourcing.
After creating an account, HelloCrowd tutorial users will get their feet wet by posting three simple tasks to the crowd platform of their choice. In addition to the “Hello, World” task above, we chose two common crowdsourcing tasks: image labeling and information retrieval from the web. In the first task, workers provide a label for an image of a fruit, and in the second, workers must find the phone number for a restaurant. These tasks can be reused and posted to any crowd platform you like; we provide simple instructions for some common platforms. The interactive tutorial will auto-generate the task urls for each tutorial user and for each platform.
More than just another tutorial on “how to post tasks to MTurk”, our goal with Hello Crowd is to teach fundamental concepts. After posting tasks, new crowdsourcers will learn how to interpret their results (and get even better results next time). For example: what concepts might the new crowdsourcer learn from the results for the “hello world” task or for the business phone number task? Phone numbers are simple, right? What about “867-5309” vs “555.867.5309” vs “+1 (555) 867 5309”? Our goal is to get new users of these tools up to speed about how to get good results: form validation (or not), redundancy, task instructions, etc.
In addition to teaching new crowdsourcers how to crowdsource, our tutorial system will be collecting a longitudinal, cross-platform dataset of crowd responses. Each person who completes the tutorial will have “their” set of worker responses to the standard tasks, and these are all added together into a public dataset that will be available for future research on timing, speed, accuracy and cost.
We’re very proud of HelloCrowd, and hope you’ll consider giving our tutorial a try.
Christian M. Adriano, Donald Bren School, University of California, Irvine
Juho Kim, MIT CSAIL
Anand Kulkarni, MobileWorks
Andy Schriner, University of Cincinnati
Paul Zachary, Department of Political Science, University of California, San Diego