  Extensive work has argued in favour of paying crowd workers a wage that is at
least equivalent to the U.S. federal minimum wage. Meanwhile, research on
collecting high quality annotations suggests using a qualification that
requires workers to have previously completed a certain number of tasks. If
most requesters who pay fairly require workers to have completed a large number
of tasks already then workers need to complete a substantial amount of poorly
paid work before they can earn a fair wage. Through analysis of worker
discussions and guidance for researchers, we estimate that workers spend
approximately 2.25 months of full time effort on poorly paid tasks in order to
get the qualifications needed for better paid tasks. We discuss alternatives to
this qualification and conduct a study of the correlation between
qualifications and work quality on two NLP tasks. We find that it is possible
to reduce the burden on workers while still collecting high quality data.
