Design Activism for Minimum Wage Crowd Work
\alignauthor Akash Mankar
\affaddr{National Instruments
Entry-level crowd work is often reported to pay less than minimum wage. While this may be appropriate or even necessary, due to various legal, economic, and pragmatic factors, some Requesters and workers continue to question this status quo. To promote further discussion on the issue, we asked Requesters and workers if they would support restricting tasks to require minimum wage pay. As a form of {\em design activism, we confronted workers with this question more directly by posting a dummy Mechanical Turk task which told them that they could not work on it because it paid less than their {\em local minimum wage, and we invited their feedback. Strikingly, for those workers expressing an opinion, two-thirds of Indians favored the policy while two-thirds of Americans opposed it. Though a majority of Requesters supported minimum wage pay, only 20\% would enforce it. To further empower Requesters, and to ensure that effort or ignorance are not barriers to change, we provide a simple public API\footnote{\url{https://github.com/akash-mankar/DesignActivism to make it easy to find a worker's local minimum wage by his/her IP address. A short version of this article appeared\,at \sc{aaai hcomp 2017\footnote{\scriptsize \url{http://www.humancomputation.com/2017/.
cscw14_minwage.tex
\documentclass{sigchi}

% Use this command to override the default ACM copyright statement (e.g. for preprints). 
% Consult the conference website for the camera-ready copyright statement.
% \toappear{
% Permission to make digital or hard copies of all or part of this
% work for personal or classroom use is granted without fee provided that 
% copies are not made or distributed for profit or commercial advantage and
% that copies bear this notice and the full citation on the first page. To
% copy otherwise, or republish, to post on servers or to redistribute to 
% lists, requires prior specific permission and/or a fee.\\
% {\confname{CSCW'15}}, Mar 14-18, 2015, Vancouver, BC, Canada.\\
% }

% Arabic page numbers for submission. 
% Remove this line to eliminate page numbers for the camera ready copy
\pagenumbering{arabic}


% Load basic packages
\usepackage{balance}  % to better equalize the last page
\usepackage{graphics} % for EPS, load graphicx instead
\usepackage{times}    % comment if you want LaTeX's default font
\usepackage{url}      % llt: nicely formatted URLs
\usepackage{ifpdf}

% llt: Define a global style for URLs, rather that the default one
\makeatletter
\def\url@leostyle{%
  \@ifundefined{selectfont}{\def\UrlFont{\sf}}{\def\UrlFont{\small\bf\ttfamily}}}
\makeatother
\urlstyle{leo}


% To make various LaTeX processors do the right thing with page size.
\def\pprw{8.5in}
\def\pprh{11in}
\special{papersize=\pprw,\pprh}
\setlength{\paperwidth}{\pprw}
\setlength{\paperheight}{\pprh}
\ifpdf
\setlength{\pdfpagewidth}{\pprw}
\setlength{\pdfpageheight}{\pprh}
\fi

% Make sure hyperref comes last of your loaded packages, 
% to give it a fighting chance of not being over-written, 
% since its job is to redefine many LaTeX commands.
\ifpdf
\usepackage[pdftex]{hyperref}
\hypersetup{
pdftitle={SIGCHI Conference Proceedings Format},
pdfauthor={LaTeX},
pdfkeywords={SIGCHI, proceedings, archival format},
pdfstartview={FitH},
bookmarksnumbered,
colorlinks,
citecolor=black,
filecolor=black,
linkcolor=black,
urlcolor=black,
breaklinks=true,
}
\else
\usepackage{hyperref}
\hypersetup{
bookmarksnumbered,
colorlinks,
citecolor=black,
filecolor=black,
linkcolor=black,
urlcolor=black,
breaklinks=true,
}
\fi

% create a shortcut to typeset table headings
\newcommand\tabhead[1]{\small\textbf{#1}}


% End of preamble. Here it comes the document.
\begin{document}

\title{Design Activism for Minimum Wage Crowd Work}

\numberofauthors{3}
\author{
 \alignauthor Akash Mankar\\
   \affaddr{National Instruments}\\
   \affaddr{Austin, TX USA}\\
   \email{akash.cs@utexas.edu}\\
 \alignauthor Riddhi J Shah\\
   \affaddr{Rackspace}\\
  \affaddr{Austin, TX USA}\\
   \email{ridhi.j.shah@gmail.com}\\
 \alignauthor Matthew Lease\\
   \affaddr{School of Information}\\
   \affaddr{University of Texas at Austin}\\
   \email{ml@utexas.edu}\\
}

\maketitle

\begin{abstract}
Entry-level crowd work is often reported to pay less than minimum wage. While this may be appropriate or even necessary, due to various legal, economic, and pragmatic factors, some Requesters and workers continue to question this status quo.  To promote further discussion on the issue, we asked Requesters and workers if they would support restricting tasks to require minimum wage pay.  As a form of {\em design activism}, we confronted workers with this question more directly by posting a dummy Mechanical Turk task which told them that they could not work on it because it paid less than their {\em local} minimum wage, and we invited their feedback. Strikingly, for those workers expressing an opinion, two-thirds of Indians favored the policy while two-thirds of Americans opposed it. Though a majority of Requesters supported minimum wage pay, only 20\% would enforce it. To further empower Requesters, and to ensure that effort or ignorance are not barriers to change, we provide a simple public API\footnote{\url{https://github.com/akash-mankar/DesignActivism}} to make it easy to find a worker's local minimum wage by his/her IP address. A short version of this article appeared\,at \sc{aaai hcomp} 2017\footnote{\scriptsize \url{http://www.humancomputation.com/2017/}}.

\end{abstract}

\keywords {
Crowdsourcing; Economics; AMT; Social Activism
}

% ML: no longer required
%\category{K.4.2.}{Computers and Society}{Social Issues}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Introduction}

Crowdsourcing is driving a new generation of intelligent systems for solving tasks that could not be effectively automated in the past~\cite{Kittur13} and powering new scales of human subjects research~\cite{mason2012conducting}. While a great deal of research has focused on optimizing technological approaches to improving work quality, affordability and speed, fewer studies have wrestled with questions of what constitutes ethical, legal, and sustainable 
practices in online crowdsourcing (cf.~\cite{felstiner2010working,irani,Kittur13,silberman}). 

For Amazon Mechanical Turk (AMT), while some workers report accepting tasks for disposable income or to pass the time, Ross et al.~\cite{Ross:2010:CSD:1753846.1753873} estimated that nearly 27\% of Indian and 14\% of American workers were dependent on AMT income for basic needs. Probing workers' {\em reservation wage} (the smallest amount a worker is willing to accept), Horton et al.~\cite{horton} found an AMT median value of \$1.38/hour.  In the US, while federal minimum wage is \$7.25/hour~\cite{wikipedia}, classification of crowd work as independent contracting excludes it from employment regulation~\cite{felstiner2010working}. Moreover, with greatly varying local minimum wages (e.g., only \$0.28/hour in India~\cite{wikipedia}) and outsourcing practices long-established, are notions of {\em equal pay for equal work} simply antiquated in a 21st century, global and digital economy?  Ethical pay rates for global participants in online studies is also unclear~\cite{mason2012conducting}.

\begin{figure}[ht]
\centering
\ifpdf
\includegraphics[width=0.5\columnwidth]{images/blakefallconroy.png}
\fi
\caption{Blake Fall-Conroy's {\em Minimum Wage Machine}.}
\label{fig:machine}
\end{figure}
As {\em design activism}, Turkopticon was deployed to both provide functional enhancements on AMT, helping mitigate power disparities, but also to provoke discussion and disrupt common rhetoric of technocentric optimism surrounding crowd-powered systems~\cite{irani}.  Blake Fall-Conroy's {\em Minimum Wage Machine} (\url{blakefallconroy.com/18.html}) provides a similar visual and experiential opportunity to try offline minimum wage work; turning the mechanical crank will yield one penny every 5 seconds, yielding \$7.25/hour (Figure~\ref{fig:machine}). 

While one can enumerate many potential technological, economic, and legal barriers to any proposed change, we might begin first by simply imagining and debating ideas for what a better model of paid crowd work might look like~\cite{Kittur13}. To seed such community {\em ideation}, let us begin by imagining a reality in which paid crowd work were required to compensate workers according to local minimum wage laws.  Would (any) Requesters be in favor of such a policy?  Would (any) workers support being blocked from available tasks that did not meet this wage restriction in their local region?  Would workers in different regions have different views?

Toward this end, we conduct a simple survey of AMT workers and Requesters (we could not survey researchers without sacrificing our anonymity, but plan to do so). To be even more provocative, we created a dummy Mechanical Turk task in which workers were told after accepting the task that they could not work on it because it paid below the worker's {\em local} minimum wage, at which point we invited their feedback (and offered payment for feedback in lieu of the task). Strikingly, for those workers expressing an opinion, two-thirds of Indians favored the policy while two-thirds of Americans opposed it. Though a majority of Requesters supported minimum wage pay, only 20\% reported willingness to enforce it. 

To empower any Requesters favoring the policy, %known or unknown, 
and to further ensure effort or ignorance are not barriers to change, we provide a simple public API to support easily finding each worker's local minimum wage by IP address. Just as Apple iTunes made it easier for people to buy music legally, can technological convenience alone yield higher pay on AMT? %engender


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Related Work}  \label{relwork} 

Several researchers have analyzed different forms of virtual work including crowdsourcing and whether they were entitled to minimum payment standards mandated under the FLSA (Fair Labor Standards Act)~\cite{flsa,felstiner2010working}. %,wolfson}. ### ML add later 
Worshall~\cite{worshall} claims that it is only fair and ethical to cover the workers by federal minimum-wage legislation. Dolan \cite{dolan} describes how Samasource uses microwork to award fair wages to poor workers in countries with a lower minimum wage. 

Silberman et al.~\cite{silberman} presented some of the first commentary on {\em worker invisibility} and ethics considerations. Martin et al.~\cite{martin} recently surveyed and interviewed workers regarding their expectations and opinions regarding crowd work. Kaufmann et al.~\cite{kaufmann} investigated worker motivations in crowdsourcing, with focus on AMT given the typical payments offered are relatively low. They tested their model by surveying 431 AMT workers. Complementing Ross et al.'s study of demographics~\cite{Ross:2010:CSD:1753846.1753873}, Ipeirotis~\cite{demog} found that 50\% of the AMT workforce was American and 40\% was Indian.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Methodology} \label{meth}

We first issued unpaid surveys for Requesters and workers on TurkerNation (\url{www.turkernation.com}), TurkerReddit (\url{www.reddit.com/r/mturk}), and on AMT itself. These surveys tended to receive only minimal response, especially for workers. For brevity, we exclude from further discussion and analysis the handful of unpaid worker responses received.  However, the unpaid AMT Requester survey did receive many responses, as summarized in Table~\ref{tab:table2}. As a known limitation of our methodology, a survey of Requesters as an AMT task is admittedly more likely to be found by workers than Requesters. More generally, we cannot verify that responses actually come from Requesters, and so workers posing as Requesters could impact our tallies. 

{\small
{\bf Question to workers:} \emph{Would you prefer it if you were allowed to work only on those tasks whose estimated reward was equal to or more than the minimum wage of your country, and be blocked from working on other lower paid tasks? Please give your opinion in at least 3 sentences.}

%\vspace{5pt}
{\bf Requester Survey}
%{\small
\begin{enumerate}
\item If you are a requester on a crowd-sourcing platform,would you prefer to reward the worker as per the minimum wage norms of the country the worker belongs to?  (Yes/No/Neutral)
\item If yes, why? Do you think you would get quality results if you did so?
\item If yes, would you block the workers whose minimum wage is greater than what you can offer per person? Why or why not?
\item If no, what are the reasons why you would not prefer this policy? Please state it clearly in 2-3 sentences.
\item Any other comments about the policy?
\end{enumerate}
}
Beyond soliciting such ``arm-chair'' opinions, we also experiment with real-time simulation of the proposed policy by putting workers into a real-life situation in which this policy is enforced. We created attractive tasks on MTurk which ask users to go to a particular link/website to gather certain information, e.g., entering restaurant information by visiting a particular link included in our HIT. In fact, the link actually points to our own website hosted on Google App Engine. % (\url{appengine.google.com}). 
Java and JSP (Java Server Pages) are used for back-end and front-end design of the website respectively. When the worker visits our website for gathering restaurant information, we lookup the worker's geographic region via IP address using an open source API called GeoIP (\url{freegeoip.net}).

In a real implementation, % of the policy, 
we would imagine that a requester could automatically lookup the appropriate local minimum wage for that worker's region~\cite{wikipedia}. The worker could be restricted from working on the task if it did not pay at least that hourly wage (assuming an estimated average task duration, perhaps by initial guestimate and/or historical task times).

For our experimental data collection, however, we simply told every worker that the task paid too little for them to work on it.  To support believability, we posted different paying versions of the same task for Indian and American workers, with each task restricted to the corresponding region (the minimum wage in India is only \$0.28 per hour). Given that AMT pays local currency in only these regions, we focused on only them for this study; future work might consider additional regions as well. Because we had separate restricted tasks for each region, this also let us easily analyze responses by region.  %Results appear in Table~\ref{tab:table1}.

Our website displayed an apology, explained the policy to workers, and then invited their feedback, offering one-time payment for that feedback instead. Workers who provided feedback were provided payment in lieu of the actual task.

%a unique code they could then enter into the AMT HIT to receive payment. 

{\small
%Workers visiting our dummy task website saw the following:

%When the worker is routed to a different website, a similar message could be shown:
%\begin{center}
\emph{We see you are located in XXX where the local minimum wage is YYY (Source:Wikipedia), and unfortunately the expected completion time for this task would mean you would be paid less than minimum wage. We are not comfortable with that, and cannot afford to pay more, and so are not allowing this task to be performed in your region. We apologize for the inconvenience and welcome your feedback. Do you think this is a good policy that we do not allow our Mechanical Turk workers to work for less than minimum wage, even though that means some tasks we post will be unavailable to workers in certain regions? %} 
%\end{center}
%In order to assure the worker the specified reward, a similar message can be displayed on the website to encourage good feedback.
%\begin{center}
%\emph{
To be fair, if you complete this feedback form and submit it, on the next page, 
you'll be able to find a unique number, which you can still use to complete this HIT and we would accept your task and pay you the amount we promised for this HIT on this form completion. Just copy paste it in the unique code section text box and submit the HIT.}
%\end{center}
}

To deter any trivial spam or carelessness, the feedback form was scripted to require completion of all fields. We later report number of visits vs.\ feedback responses received. %measured many more visits than responses, indicating many workers decided not to provide feedback. %Requiring all fields may also have prevented some spam.


Workers were required to have previously completed 100 HITs in order to access our tasks. We did not require any prior approval rate. Inspired by {\em Turkalytics}~\cite{heymann2011turkalytics}, we also used IP tracking to check if we received visits from the same IP address for different worker IDs, or the same worker ID for different IP addresses. Beyond fraud, this might arise in benign settings like an internet cafe with dynamic IP addressing. 
%While our system was designed to display an error webpage in this event, 
We detected no such occurrence in practice.  A technological hurdle we could imagine in practice with such a policy is IP-spoofing: if pay is region dependent, one might try to change one's IP to appear to be located in a higher-paying region.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Results} \label{results}

Worker responses are summarized in Table~\ref{tab:table1}. While paid AMT survey responses show workers (regions unknown) as evenly divided on the policy, worker feedback from our AMT dummy task was quite different. 241 of 301 US workers who visited our website left feedback.  Interestingly, we received far fewer Indian visits to our website: 104, with 62 responses. Far more strikingly, the distribution of opinions between regions diametrically-opposed, with two-thirds of Indian workers expressing an opinion in favor of the policy, while two-thirds of such American workers oppose the policy. %These diametrically opposing responses are shown visually in Figure \ref{fig:UUSINDworker}. 

\begin{table}[ht]
\centering
\caption{Worker Survey Results show opposite USA and India responses.}
\begin{tabular}{|l|c|rr|rr|rr|}
\hline
Region & Total & \multicolumn{2}{c|}{Yes} & \multicolumn{2}{c|}{No} & \multicolumn{2}{c|}{Neutral} \\ 
\hline
\multicolumn{8}{|c|}{{\footnotesize\em Paid Survey on Mechanical Turk}} \\
\hline
All & 73 & 32 & 44\% & 33 & 45\% & 8 & 11\%\\  
\hline
\multicolumn{8}{|c|}{{\footnotesize\em Feedback on Mechanical Turk Posted Task}} \\
\hline
USA   & 241 & 55 & {\bf 23\%} & 121 & {\bf 50\%} & 65 & 27\%\\
India & ~~63& 31 & {\bf 49\%} &  15 & {\bf 24\%} & 17 & 27\%\\
All   & 304 & 86 & 28\% & 136 & 45\% & 82 & 27\%\\
\hline
\multicolumn{8}{|c|}{{\footnotesize\em Combined Results}} \\
\hline
All & 377 & 118 & 31\% & 169 & 45\% & 90 & 24\%\\
\hline
\end{tabular}
\label{tab:table1}
\end{table}

Requester feedback is summarized in Table~\ref{tab:table2}. We received a total of 88 responses (82 from AMT, and only 6 from both TurkerNation and TurkerReddit). Across all responses, a surprising majority 58\% of Requesters report favoring minimum wage payment, with 39\% opposed (Question 1). Of those in favor, 71\% expected better quality work would result from such a payment policy, with only 1\% opposed (Question 2). Despite this, only 20\% of Requesters indicated willingness to block workers from performing tasks that would underpay vs.\ local minimum wage laws: 35\% opposed the policy, while a large 45\% of Requesters expressed neutrality on the question.

%on these surveys. However, the Qualtrics survey on Turkernation and TurkerReddit had only 6 responses, so we don't draw any conclusions for such a small set of responses. On the other hand, AMT unpaid survey was completed by 82 requesters and 49 of them termed this a good policy and stated that they are more than happy to pay the minimum wage if the quality of work is going to improve. Also there is a belief that if workers are treated like full time employees with anonymity, the quality of work will improve. However, one limitation of these surveys was we clearly stated that complete the survey only if you were a requester but we didn't have a mechanism to verify whether a requester is completing the survey or a worker.

\begin{table}[ht]
\centering
\caption{Results of Requester Surveys (all unpaid).}
\begin{tabular}{|l|c|rr|rr|rr|}
\hline
\!\!Question\!\! & \!\!Total\!\! & \multicolumn{2}{c|}{Yes} & \multicolumn{2}{c|}{No} & \multicolumn{2}{c|}{Neutral\!\!} \\ 
\hline
\multicolumn{8}{|c|}{{\footnotesize\em TurkerNation \& TurkerReddit}} \\
\hline
\!\!1.\ {\footnotesize Prefer pay min-wage?}\!\!     & \!\!~6\!\! & 2\!\! & & 3\!\! & & 1\!\! & \\  
\!\!2.\ {\footnotesize Expect quality results?}\!\!  & \!\!~2\!\! & 2\!\! & & 0\!\! & & 0\!\! & \\  
\!\!3.\ {\footnotesize Block if pay too low?}\!\!    & \!\!~2\!\! & 0\!\! & & 2\!\! & & 0\!\! & \\  
\hline
\multicolumn{8}{|c|}{{\footnotesize\em Mechanical Turk}} \\
\hline
\!\!1.\ {\footnotesize Prefer pay min-wage?}\!\!    & \!\!82\!\! & 49\!\! & \!\!60\% & 31\!\! & \!\!38\% &  2\!\! &  \!\!2\%\!\!\\
\!\!2.\ {\footnotesize Expect quality results?}\!\! & \!\!49\!\! & 34\!\! & \!\!69\% &  1\!\! &  \!\!2\% & 14\!\! & \!\!29\%\!\!\\
\!\!3.\ {\footnotesize Block if pay too low?}\!\!   & \!\!49\!\! & 10\!\! & \!\!20\% & 16\!\! & \!\!33\% & 23\!\! & \!\!47\%\!\!\\
\hline
\multicolumn{8}{|c|}{{\footnotesize\em Combined}} \\
\hline
\!\!1.\ {\footnotesize Prefer pay min-wage?}\!\!    & \!\!88\!\! & 51\!\! & \!\!58\% & 34\!\! & \!\!39\% &  3\!\! &  \!\!3\%\!\!\\
\!\!2.\ {\footnotesize Expect quality results?}\!\! & \!\!51\!\! & 36\!\! & \!\!71\% &  1\!\! &  \!\!2\% & 14\!\! & \!\!27\%\!\!\\
\!\!3.\ {\footnotesize Block if pay too low?}\!\!   & \!\!51\!\! & 10\!\! & \!\!20\% & 18\!\! & \!\!35\% & 23\!\! & \!\!45\%\!\!\\
\hline
\end{tabular}
\label{tab:table2}
\vspace{-5pt}
\end{table}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Worker Opinions}

In addition to the above tabular data, it was also interesting to note the diversity of open-ended comments we received.  We select a few of these comments to include here as illustrative. The entire dateset is publicly accessible$^\textnormal{1}$. 
%Following blind review, the entire dataset will be made public (redacted for respondent-identifying information).

%As mentioned earlier we received a diverse set of responses, with opinions being skewed in favor of the minimum wage restriction policy or against it based on the location of the worker for Experiment 2.
%Below we mention and summarize some selected subset\footnote{the entire dataset will be made available publicly on web soon} of contrasting opinions.

\subsection{Supporting Opinions}
\emph{``It is a good policy because it shows you care about the turkers. Also it stops us from wasting our time. This way more people can take part in surveys.''} 

\emph{``\ldots this is a pretty good policy because if you make over a certain amount of dollars on mturk you have to file that information in your taxes. If I am being taxed on wages earned on mturk then shouldn't the minimum wage law apply too?''}

\emph{``This is fantastic because most requesters are fine with having workers work for far less than minimum wage.''}

A recurring theme among positive responses is a sense that being able to get paid minimum wage shows appropriate respect towards workers as a task force. While some workers express concerns that Requesters are often unfair in terms of wage, such a policy might bring greater fairness and equality to all parties.  As shown above, some express a belief that taxation should entitle minimum wage guarantees.

%concerns being taxed without of paying the minimum wage as the amount they earn is taxable. 

\subsection{Neutral Opinions}
\emph{``I think it is a nice idea but this is something that many people choose to do in their free time and I don't think you need to pay minimum wage.''}

\emph{``I like it because I think it's a very noble idea. The MTurk site truly does not pay very much. However, a lot of us are here because we really need money, and the small amounts add up to something at the end of the month.''}

%Workers who had neutral opinion about the policy talked about pros and cons of the policy, while 
Some workers in this group suggested that this policy seems ideal but not realistic, while others noted AMT is only contract work and something done only for supplemental income. 

%is an ideal policy but not realistic, other feel Mturk workers are contract workers and MTurk is something that they do to make quick money in less time, but do not rely on it for their daily living. While this is true for US based workers, the same opinions comes as a contrast for Indian workers as some of them do rely on MTurk for their daily living.

\subsection{Opposing Opinions}
\emph{``If this were not the policy I could still make \$5.00 which I would be fine with and instead I am getting a lot less money than that because I cannot work on the hit. Therefore, I find this policy a bad policy.''}

%\emph{``There are folks out there who are unable to work in a typical scenario. These kinds of jobs help them feel like they contribute to their well being. I am one such person and this supplements my disability income\ldots''} % as well as the previous stated benefit.''}

\emph{``Because I am working from home a few minutes at a time, I don't believe this type of work should come under the fair wage and hour laws. I am making a conscious decision to work on the pay scale provided, although it is low, which I believe is my [right]\ldots''} % [...rest of the post omitted...]''}

\emph{``This is a bad policy. Minimum wage does not apply to contract workers. Give me my HIT!!!''} 

\emph{``If I'm willing to work for less, I should have the option. I don't think it's up to someone else to decide my fair wage. Besides, my completion time may be quicker, resulting in a higher pay/hour.''}

Workers against the policy expressed firm beliefs that they are capable of making their own decision about whether to attempt a given task. In some cases, workers are happy getting paid less than minimum wage because the amount is still significant to them, and such a policy could prevent this. %them from earning even this amount. 


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Requester Opinions}

\subsection{Supporting Opinions}
\emph{``The minimum wage seems like the most fair way to reward people. I would not exclude people who usually make a higher minimum as long as they know what I was offering.''}

As noted earlier, a majority of Requesters supported the minimum wage policy. Their comments suggest a firm belief that it is a fair way to reward people working on AMT tasks. Some of these opinions were argued based on standards and ethics. % that should be followed in crowdsourcing.

\subsection{Neutral opinions}
\emph{``It depends on the task and the difficulty involved I wouldn't block someone, unless they broke the rules I set up''} 

Few of the opinions were neutral, but interesting. They suggested that these Requesters were fine with paying the minimum wage assuming quality improvement justified it. However, they did not want to block anyone.

\subsection{Opposing Opinions}
\emph{``It is unfair to give people minimum wage [by] country when this is a crowd sourced platform where people come from different parts of the world. They deserve to be treated equally.''} % based on 

Such requesters suggested that internet as a whole is one entity, and that people are ``netizens'' of the Internet and not citizens of any particular country. In the spirit of {\em equal pay for equal work}, these comments suggested all netizens should be treated and paid equally, irrespective of physical location.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Further Analysis of Feedback}
In reviewing opinions we received, we further organized them into emergent categories to identify recurring themes and their relative distributions. Results appear in Table~\ref{tab:table4}.
%it is imperative to classify them in further sub-categories rather than just sticking to good, bad and neutral opinions. We sub-categorize only worker and Requester opinions which are in favor of the policy and which oppose it. 
We exclude neutral opinions, which did not seem to have any distinctive categories. The most common reason for a neutral opinion for both Requesters and workers was principally that their opinion depends on the type of task at question. 

\begin{table}[ht]
\small
\centering
\caption{Sub-categorization of worker and Requester opinions}
\begin{tabular}{|l|l|c|}
\hline
\hline
                                \multicolumn{3}{|c|}{\textbf{Workers}}                                                                              \\ \hline
                                & \multicolumn{1}{c|}{\textit{\textbf{In Favor of the Policy}}}                                                      & \textit{\textbf{118}} \\ \hline
1                               & Ethically Fair                                                                                                     & 42                    \\ \hline
2                               & Legally Compliant                                                                                                  & 8                     \\ \hline
3                               & Help them find higher paying work more quickly                                                                     & 8                     \\ \hline
4                               & Encourages them to do better work                                                                                  & 4                     \\ \hline
5                               & Will help meet their basic end needs                                                                               & 9                     \\ \hline
6                               & Miscellaneous                                                                                                      & 47                    \\ \hline
\multicolumn{1}{|c|}{\textit{}} & \multicolumn{1}{c|}{\textit{\textbf{Against the Policy}}}                                                          & \textit{\textbf{169}} \\ \hline
1                               & \begin{tabular}[c]{@{}l@{}}Ethically Unfair  (performance/merit not location \\ and its minimum wage)\end{tabular} & 11                    \\ \hline
2                               & Opportunities reduce                                                                                               & 18                    \\ \hline
3                               & It is not a MTurk/crowdsourcing suitable policy                                                                    & 19                    \\ \hline
4                               & Not a source of basic income                                                                                       & 14                    \\ \hline
5                               & Workers capable of making a conscious decision                                                                     & 71                    \\ \hline
6                               & Not here for money only/Like working on tasks                                                                      & 9                     \\ \hline
7                               & Miscellaneous                                                                                                      & 27                    \\ 
\hline 
\hline
\multicolumn{3}{c}{}\\
\hline
\hline

                                \multicolumn{3}{|c|}{\textbf{Requesters}}\\ \hline
                                & \multicolumn{1}{c|}{\textit{\textbf{In Favor of the Policy}}}                                                      & \textit{\textbf{51}}  \\ \hline
\multicolumn{1}{|c|}{1}         & Incentive for quality work                                                                                          & 36                    \\ \hline
\multicolumn{1}{|c|}{2}         & Ethically fair                                                                                                     & 8                     \\ \hline
\multicolumn{1}{|c|}{3}         & Attracts more workers                                                                                              & 6                     \\ \hline
\multicolumn{1}{|c|}{4}         & Miscellaneous                                                                                                      & 1                     \\ \hline
                                & \multicolumn{1}{c|}{\textit{\textbf{Against the Policy}}}                                                          & \textit{\textbf{34}}  \\ \hline
\multicolumn{1}{|c|}{1}         & Not a MTurk/crowdsourcing suitable policy                                                                          & 5                     \\ \hline
\multicolumn{1}{|c|}{2}         & Ethically unfair                                                                                                   & 10                    \\ \hline
\multicolumn{1}{|c|}{3}         & Work is low skill/low effort, not worth minimum wage                                                               & 8                     \\ \hline
\multicolumn{1}{|c|}{4}         & Cannot afford or lack of funding                                                                                   & 2                     \\ \hline
\multicolumn{1}{|c|}{5}         & Do not want to spend (more productive / business model)                                                            & 2                     \\ \hline
\multicolumn{1}{|c|}{6}         & Miscellaneous                                                                                                      & 7                     \\ \hline
\hline
\end{tabular}
\label{tab:table4}
\end{table}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Empowering Requesters} \label{emp}

As noted earlier, a majority of Requesters supported our posited minimum wage policy. In order to empower this section of the Requester community, we created a simple Python API$^\textnormal{1}$
%\footnote{\url{https://github.com/akash-mankar/DesignActivism}} 
which can be used to help implement this policy in practice. %\footnote{For anonymity, our Github API URL will be given after review.}.
Given the worker's IP address, the Requester can first lookup the worker's region via the aforementioned open-source GeoIP library. Next, the Requester uses our API to automatically look up the local minimum wage for that region, using rates from Wikipedia~\cite{wikipedia}. 


%Figure \ref{fig:api} depicts the workflow for requesters and how the API call can be done during HIT creation. While creating a Mturk task using the User Interface or the MTurk SDK, %(Software Development Kit)\footnote{\url{http://aws.amazon.com/developertools/Amazon-Mechanical-Turk/694}}, 
%the reward per HIT is specified within the HIT properties. Before populating this field, an independent call to our Python Github API can give the requester an idea about the minimum wage based payment threshold. The API obtains the location specific minimum wage as per Wikipedia\footnote{\scriptsize\tt{en.wikipedia.org/wiki/List\_of\_minimum\_wages\_by\_country}}. In order to be able to use this API, an installation of Python2.7. % or a greater version\footnote{\url{https://www.python.org/downloads/}}is required.

%\subsubsection{Example usage}
As an example, imagine a Requester wants to publish a task in India with 10 HITs and allocates 6 minutes for the completion of each HIT (60 minutes for all the HITs). Before specifying the reward, the requester runs the python program and provides the location (India) to it as a command line argument. This returns the minimum wage/hour in US\$ for the country India (\$0.28/hour). If the requester wants to enforce the minimum wage policy, he/she has to make sure the estimated net reward for the task is greater than \$0.28/hour, i.e., at least \$0.028 per HIT. For a general task available globally, the Requester could dynamically detect the worker's region and either block the worker from performing the task, or provide bonus payment up to the local minimum wage rate.

%The requester can then publish the HITs and analyze apsects like result quality and response rate. 

%\begin{figure}[!h]
%\centering
%\ifpdf
%\includegraphics[width=1.0\columnwidth,totalheight=0.3\textheight]{api}
%\fi
%\caption{Requester workflow} 
%%\midskip
%\vspace{5pt}
%\small{Depicts API usage during HIT creation}
%\label{fig:api}
%\end{figure}

\section{Conclusion and Future Work} \label{conc}

We conducted this study without an intent to insist that our ``straw-hat'' policy would be an ideal and universal solution to fairness in crowdsourcing. Instead, we built a system whose results are useful to make worker---requester relationship more transparent and to provoke the thoughts of globally situated workers and Requesters for an ethical debate regarding minimum wage. The responses that we received reflect interesting patterns of thought for engaging dialog and informing researchers and others seeking to imagine alternative future designs for ethical and sustainable crowdsourcing.

%This study tried to probe worker and requester opinions globally to provoke the debate of fair wage in the crowdsourcing community. We sought requester and worker perceptions, if the aspect of minimum wage is incorporated into crowdsourcing platforms.  We gathered answers to our research questions from the workers' as well as requesters' perspectives. The study shows interesting patterns and divide of opinions among workers as well as requesters. This research establishes that, for more than 90\% of MTurk population situated in India and United States alone, based on the location of the worker it can be decided whether to enforce minimum wage policy or not. Also majority of requesters are willing to pay the minimum wage provided quality of work improves. This paradigm can be used for locations like India which are pro minimum wage policy and at the same time use a very nominal minimum wage compared to US minimum wage. We propose that the prospect of minimum wage policy is an exciting opportunity for the crowdsourcing community. The results of this study will be instrumental for spreading awareness in the crowdsourcing community about expectations of workers and requesters from each other and can make worker-requester relationship more transparent.

We would like to advertise our API to, and solicit feedback from, the research community, to further our own understanding and to promote ongoing dialog.

\section*{Acknowledgements} 

We thank our talented crowd workers, without which our study would not be possible, as well as fellow researchers who provided valuable feedback in preparing this paper. This study was supported in part by National Science Foundation grant No. 1253413 and IMLS grant RE-04-13-0042-13. Any opinions, findings, and conclusions or recommendations expressed by the authors are entirely their own and do not represent those of the sponsoring agencies.

%Following blind-review, we plan to both advertise our API to, and solicit feedback from, the research community, to further our own understanding and to promote ongoing dialog.
%In future work, we plan to advertize the Github API to empower requesters who are mostly researchers so that we can gather deeper insight into the perspectives of the research community. 
%{\small

\bibliographystyle{acm-sigchi}
\bibliography{sample}
%}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section{Appendix}
%\normalfont

%The following section describes a few model tasks that can be created in order to be able to reproduce the results. The templates can be modified as per the experiment being conducted.

%{\bf Worker question surveyed:} %A few sample questions:
%\begin{enumerate}
%\item As a crowdsourcing worker, would you prefer it if you were notified about tasks whose estimated reward was equal to or more than the minimum wage of your country? (Yes/No/Neutral)
%\item If yes, 
%\item If neutral, why? Please give your opinion in at least 3 sentences.
%\end{enumerate}


%\subsection{Real-time simulation of the policy}




%Figures \ref{fig:feedback}, \ref{fig:ty}, \ref{fig:error} depict sample layouts for the implementation of this task.
%
%
%\begin{figure}[!h]
%\centering
%\ifpdf
%\includegraphics[width=0.9\columnwidth]{feedback}
%\fi
%\caption{Layout for a feedback form page. Also notice the location detected as (city, state, country).}
%\label{fig:feedback}
%\end{figure}
%
%\begin{figure}[!h]
%\centering
%\ifpdf
%\includegraphics[width=0.9\columnwidth]{ty}
%\fi
%\caption{Layout for a response page showing a unique code which the worker could use to submit the HIT.}
%\label{fig:ty}
%\end{figure}
%
%
%\begin{figure}[!h]
%\centering
%\ifpdf
%\includegraphics[width=0.9\columnwidth]{error}
%\fi
%\caption{An example error message when a fraudulent worker visit is detected}
%\label{fig:error}
%\end{figure}


\end{document}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


% Abstract
%
%Crowdsourcing platforms such as Amazon Mechanical Turk (AMT/MTurk) help task requesters reach out to a large number of freelance employees that include low income workers from developing countries as well as workers in developed countries. In both cases, there is always a possibility that workers do not get the minimum compensation they deserve according to the respective country's norms. We intend to introduce social activism in crowdsourcing addressing the same issue, thus making workers as well as requesters aware of the minimum wage that needs to be paid according to the location of the worker. The main purpose of this study was to provoke thoughts of the crowdsourcing community globally about the fair wage policy and addressing concerns of the workers and requesters in this perspective. The main contribution of this study is that it will be instrumental in increasing the transparency between worker---requester relationship and helpful in spreading awareness about expectations of workers and requesters from each other. We gathered and analyzed requester and worker perceptions and as per our conclusions we claim that requesters can make a more informed decision about enforcing the minimum wage policy based on the location of the worker in order to produce high quality results. 


%    We propose a study in which we try to understand the impact of incorporating the minimum wage factor in crowdsourcing tasks. By means of this study, we seek answers to two important research questions mentioned below:
%
%
%\begin{itemize}
%
%\item 
%What is the perspective of workers when they are not allowed to work on tasks where wage/hour is less than minimum wage/hour
%in their country/location? Do they prefer it that way, or are they interested in working on tasks which are blocked for them? If so, why?
%
%\item
%Will the requesters move to crowdsourcing the tasks only to a category of countries where the workforce feels they are being paid substantially and hence will deliver quality work, or do they feel that a task can achieve good results and gather a workforce even in developed countries with a higher wage bar?
%\end{itemize}
%
%Overall, our goal is to understand how the minimum wage factor probes the crowdsourcing community and if such social activism would be well received by the people in this community. 
%\\
%\\We expect to have a diverse set of opinions and answers to these questions. The general opinion is that, requesters, depending upon what they give more weight to - cost or result quality, would be for or against this idea respectively. Similarly, workers would generally be excited at the proposition of getting paid well and avoid working on tasks that would pay them less. However, there may not be enough tasks which follow minimum wage norms. In such a case, restricting a worker from taking up tasks can have a counter effect too. Many workers could be on platforms like AMT because they like working on a variety of tasks and not for the money.It would be interesting to get an idea of all these view points and explore the overall dynamics of the effect of such norms in the crowdsourcing community. We intend to come up with a study that will give us a clear picture on what the general opinion is in the crowdsourcing community regarding such social activism. Does it get provocative?
%
%As per the results obtained, this policy generated a lot of interest in the crowdsourcing community. The location of the work can be a determining factor while enforcing this policy as the workers from India, representative of regions with a low minimum wage, were in favor of the policy. On the other hand, workers from countries like the United States believed that it is an unfair policy and could lead to a reduction in the number of tasks available to them. Interestingly, requesters liked such a policy as they think it would help them obtain quality results.
%
%The paper is divided as follows: Section on Related Work briefly gives an overview of the previously done related work in this field. The Section on Methodology elaborates on the various experiments conducted for the purpose of this study. The results for the study are presented in the Results section and a relevant discussion and analysis has been done in the Discussion section. The conclusion and future work are outlined in the concluding section. 

%owever, instead of using such third party channels, our study focuses on similar distribution of work being incorporated into MTurk itself.

%The work provides some analysis on the reasons behind the wages being drastically below the minimum wage and the demand supply scenario in the crowdsourcing community that contributes to this. Ross et al.\cite{Ross:2010:CSD:1753846.1753873} elaborated on the diversity across several notable demographic dimensions of the MTurk workers that included income. The most notable point outlined in the paper was the percentage of the crowd who is dependent on this platform for basic end needs (14\% of United States Turkers, 27\% Indian Turkers, 15\% Turkers always or sometimes). This is quite a motivation to think if the workers are, in this scenario, being paid at least the minimum wage. And if they are not, would the Turkers still benefit? Due to the dependence of a large number of workers on MTurk, in a Forbes article by 
%He says,
%\begin{center}
%\emph{``Ever wondered what our labor market would look like without minimum wages or labor law protections? Take a look at the brave new world of online piecework platforms, like Amazon’s Mechanical Turk, which allows employers, politely termed 'requesters', to post jobs for a “global, on-demand, 24 x 7 work force.''}
%\end{center}

%Apart from this, a number of observations about the worker’s earnings targets depicted the variety of worker opinions about incentives. Our study would test if these opinions reflect in the feedback obtained for the minimum wage policy.

%In a Forbes article by Dolan \cite{dolan}, she describes how Samasource %\footnote{\url{http://samasource.org/}}, uses microwork as a successful route to award wages to poor people in countries with a lower minimum wage. However, instead of using such third party channels, our study focuses on similar distribution of work being incorporated into MTurk itself. 

%A study conducted by Martin et al. \cite{martin} called 'Being a Turker' shows various aspects of different turkers and their expectations and opinions as workers. It also affirms our emphasis on choosing TurkerNation as one of the platforms where we should seek opinions for our study.

%In our study, we intend to go a step ahead and gather perspectives from both the requesters and workers all over the world, by surveying them about the minimum wage policy and also, collaborating their feedback by trying to incorporate this policy into the crowdsourcing platform. 


%While we believed this policy would generate a great set of responses, we got very few responses and these were not very descriptive. Our analysis suggested that when we provided a paid HIT to the worker, we promised to pay them a certain amount (\$0.10 to \$0.20) on HIT completion, but when we had to get feedback on our minimum wage policy we had to restrict them from working on the HIT, which meant even after providing the feedback, they would not get paid anything. We figured this was the main reason for low responses (no reward), so we modified our HIT and on completion of the feedback on our website, generated a unique code, which if the worker enters on the HIT and submits it, he or she would get paid completely as much as the HIT promised. Also we learned from some of the results mentioned in Chilton et al. \cite{chilton2010task} on how tasks are normally searched by MTurk workers and incorporated some of the ways in order to increase visibility for our tasks on MTurk. After this, we started getting a great number of responses. 

%As mentioned earlier the site was hosted on Google App Engine as it is scalable, provides an easy deployment environment and also a datastore to record the results.Creating these `bogus' tasks meant we had to ensure that the we do not pay the worker more than the minimum wage for the HIT based on the geographic location, otherwise showing them a message of `restriction from working on a task' would have been invalid. While this was easy to achieve for countries like United States where the minimum wage per hour is \$7.25 per hour, but for developing countries like India where minimum wage is only \$0.28 per hour, we had to make sure our HIT reward stays within that minimum.

%At this point, it is also worthwhile to note that according to a study conducted by 
%
%At the time of conducting this study, the majority is reversed with worker population dominated by India and United States being a close second. Hence for experiment 2, we designed two separate HIT tasks which were only restricted to United States and India each. Since more than 90\% of the Mturk population resides within these countries we believe our results are not biased and can be generalized to all crowdworkers. However we plan to scale this to all countries in future.


%We consider this as a very important step in our study as surveys may give us information about the general inclination of the workers, but many would behave differently in a real life scenario than if they were asked what they would do in a hypothetical scenario. 


% summarizes Requester responses across surveys.
%
%Our study was majorly divided into two sections, one for gathering Worker perceptions and the other for Requester perceptions. We conducted series of surveys on different online forums for workers and requesters, including TurkerNation, TurkerReddit and also an 

%\subsection{Platform Overviews}
%This section gives a brief overview of the platforms that were instrumental to conduct the experiments for our study. These platforms have many active workers and requesters participating in them (especially Turkers), and hence were considered to be the optimal place to gather feedback on the minimum wage policy.
%
%\subsubsection{TurkerNation}
%TurkerNation (\url{www.turkernation.com}) is a volunteer run forum that helps workers and requesters figure out the most effective way of working with Mechanical Turk. It has discussions on various MTurk topics, adheres to pre-defined rules and more importantly has a live and active chatroom community that provides instant replies on a wide variety of topics and issues. 
%
%\subsubsection{TurkerReddit}
%TurkerReddit (\url{www.reddit.com/r/mturk}) 
%%is another widely used forum of Reddit\footnote{\url{http://www.reddit.com/}} members who are also Turkers. It 
%is a platform for discussion of MTurk tips and tricks scam alerts,research studies and many other useful discussions. It is an open forum where workers can post links of interesting/high-paying HITs to bring it to notice by other Turkers.

%\subsubsection{Qualtrics}
%Qualtrics %\footnote{\url{http://www.qualtrics.com/}}, 
%provides an online survey software and insight platform was also used for requester surveys as it provides a very good results aggregation feature.


%\subsection{Experiment Design}
%In order to gather feedback on the minimum wage policy from both requesters and workers, a different set of experiments have been designed for each group as shown in Figure \ref{fig:meth}. 
%
%%\begin{figure}[!h]
%%\centering
%%\ifpdf
%%\includegraphics[width=0.9\columnwidth]{meth}
%%\fi
%%\caption{Methodology and Experiments}
%%\label{fig:meth}
%%\end{figure}
%
%\subsubsection{Worker Perceptions}
%We obtained worker perceptions in two distinct ways – an explicit survey/question asking and a pseudo-implementation of such a policy on MTurk, as discussed in the following experiments.
%
%\paragraph{Experiment 1}
%First, we gathered opinions by means of a short survey explicitly questioning the crowdsourcing workers about their take on such a policy. These surveys have been posted on various open forums like TurkerNation and TurkerReddit which are frequently used by Turkers. Besides, first an unpaid and then a paid version of a survey task was also created on MTurk. We also have experimented by posting different style of questions to explore the styles that generate maximum response, hence enabling us to refine our experiments. 

%###
%The following was the question posted on TurkerReddit.
%\begin{center}
%\emph{As a crowdsourcing worker, would you prefer if you were notified about tasks whose estimated net reward was equal to or more than the minimum wage of your country? If yes, would you also prefer it if you were allowed to work only on these tasks and be blocked from working on the other lower reward tasks?}
%\end{center}
%As the above question obtained a decent number of responses, even on TurkerNation, the same question was posted in the live chatroom where the community is generally most active and also generated quick responses.  In AMT, two different tasks were created. Initially an unpaid survey task was published. This and all previous tasks were unpaid and we believe that we can solicit quality worker opinions who respond to these tasks and like the idea of continuously improving the  MTurk experience . 
%However we also wanted to seek opinions of those who do not want to complete a survey for free. So we created a paid survey (\$0.1 per HIT) with the question framed a bit differently.
%
%\paragraph{Experiment 2}
%Apart from explicitly questioning the workers about the policy, we also decided to incorporate this policy into MTurk and gather appropriate feedback. In order to this, 

%We allowed only those workers to work on our HITs who have completed more than 100 HITs in their MTurk career. This ensured we had workers with enough experience but not necessarily all who have lifetime approval rate of around 95\%. This ensured inclusion of enough experienced workers.

%\paragraph{Geographic Restriction}
%Since we wanted feedback only from India and US for this experiment we designed HITs which were restricted to these regions. This ensured that workers from other countries would not be able to complete the feedback form.

%\paragraph{Feedback form compulsory fields}
%
%\paragraph{IP detection and Worker Id based quality control}
%
%
%\subsubsection{Requester Perceptions}
%In order to gauge what the requester community feels about the minimum wage policy, we again designed a different set of experiments.
%
%\paragraph{Experiment 1}
%In the first phase, we explicitly asked the requesters for their opinion on such a policy. Similar to the worker experiments, we posted surveys on TurkerNation, TurkerReddit and MTurk (unpaid). This time around, the surveys for the former two forums were designed using the survey design tool Qualtrics (mentioned in  platform reviews section) in order to explore a new design style as this tool offers good result aggregation options. The survey link was posted on both TurkerNation and TurkerReddit. An unpaid survey was created on MTurk for the requesters.

%In TurkerNation, TurkerReddit and AMT unpaid surveys we received very few responses (less than 8 in each) and hence we have wide difference in results and can't be used to come to a conclusion. However for AMT paid surveys, we received a total of 73 responses. Out of 73, 32 responses were in favor of the minimum wage restriction policy, 33 against and 8 workers had neutral views about it. Numbers clearly shows the 50-50 divide in worker perspective. Also since these surveys were accessible all over the world, results also show that this policy does probe the crowdsourcing community in global perspective.

%\begin{table}[h]
%\begin{tabular}{|l|r|r|r|r|}
%\hline
%%  & \multicolumn{1}{l|}{Survey}          & \multicolumn{3}{c|}{\%Opinions} \\ 
%Survey Type  & \!\!Responses\!\! & \!For\!     & \!Neutral\!     & \!Against\!    \\ \hline
%TurkerNation & 5 & 4 & 0 & 1 \\ 
%TurkerReddit & 4 & 2 & 1 & 1 \\ 
%AMT unpaid   & 2 & 0 & 0 & 2 \\ 
%AMT paid     & 73 & 32 (44\%)\! & 33 (45\%)\! & 8 (11\%)\! \\ 
%\hline
%Expt2: US    & 241 & 55 & 65 & 121 \\
%Expt2: India &  62 & 31 & 17 &  14 \\
%\hline
%Total & 387 & & & \\
%\hline
%\end{tabular}

%\begin{figure}[!h]
%\centering
%\ifpdf
%\includegraphics[width=1.0\columnwidth]{UUSINDworker}
%\fi
%\caption{US vs India - Contrasting opinion trends- Experiment 2}
%\label{fig:UUSINDworker}
%\end{figure}

%Combined overall results are summarized in Figure \ref{fig:WorkerRes}. These include all the surveys mentioned in Table \ref{tab:table1}. As shown in the figure, a total of 43\% participants said it is a bad policy, most of them didn't like the idea of being blocked on the tasks. 45\% voted in favor of the policy while a small percentage of participants (12\%) believed that it is a good policy but restricting someone working on the task is not good.
%\begin{figure}[!h]
%\centering
%\ifpdf
%\includegraphics[width=0.9\columnwidth]{WorkerRes}
%\fi
%\caption{Summary of worker survey results}
%\label{fig:WorkerRes}
%\end{figure}

%The above experiment was conducted without considering the location of the worker. In the next set of results for experiment 2, we considered results based on location, to see if any interesting patterns can be seen using location as the parameter.
%
%\subsubsection{Experiment 2 Results}
%Experiment 2 was conducted both for India and United States with different HITs.
%
%\subsubsection{United States}


%50\% of the Indian workers believe that it is a good policy and minimum wage should be enforced. On the other hand, nearly equal percentage of workers believe that it is either a bad policy or it is good but restricting from task is not a solution.


%In United States, a total of 301 workers attempted to complete our HIT, but only 241 workers actually left a feedback, we assume the other workers didn't feel comfortable that they were blocked from working on the restaurant finding HIT and did not bother to complete the feedback form even when it meant they earn the same amount for filling up the form.

%Majority of the opinions we received from US workers termed this policy as bad. Out of 241, we received 121 responses against the policy, 65 had neutral opinions and couldn't decide whether they are in favor or against, while 55 US workers really liked it.

%\subsubsection{India}
%Surprisingly we had lesser responses from India as compared to United States when the majority of turker population now resides in India.  We received 62 responses out of a total 104 Indian turkers who visited our website. However, 

%We can observe clearly a trend against the policy in US while in favor of the policy in India. In fact, the numbers are exactly mirror images of each other. In both cases we have 27\% neutral opinions,  23\% against in India and in favor in US, while 50\% against in US and in favor in India. This is a very important and useful result in this research.
%
%
%\subsubsection{Geographical divide of opinions}
%%Figures \ref{fig:USA} and \ref{fig:india} depict the geographical regions from which the various responses were obtained from within US and India respectively for Expreriment 2. As observed from Figure \ref{fig:USA}, 
%Our task obtained feedback from nearly 78\% of the US states showing that the policy generated a lot of interest in the country where the minimum wage norms differ across states. On the other hand, %as per Figure \ref{fig:india}, 
%the responses were concentrated in only 31.4\% of the Indian states. Hence, the coverage of the policy in India was comparatively weaker. But this also depends on where most of the crowdsourcing population is located within India and United States. In US, it appears as if it is equally divided among almost all states where as in India participating states are more technologically aware and hence it is understandable that these states are dominating in crowdsourcing community too. 

%\begin{figure}[!h]
%\centering
%\ifpdf
%\includegraphics[width=0.9\columnwidth]{USA}
%\fi
%\caption{Geographical divide of opinions within the US}
%\label{fig:USA}
%\end{figure}
%
%\begin{figure}[!h]
%\centering
%\ifpdf
%\includegraphics[width=0.7\columnwidth]{india}
%\fi
%\caption{Geographical divide of opinions within India}
%\label{fig:india}
%\end{figure}

%If we consider locations, a little bit less than minimum wage in US is way more than minimum wage in India. One of the workers pointed out the concern of disabled workers,  and mentioned that Mturk jobs make them feel like contributing to self well-being and getting paid less is not really a concern. 

%\begin{tabular}{|l|r|r|r|r|}
%\hline
%\!\!Question\!\! & \!Total\! & \!Yes\!     & \!No\!     & \!Neutral\! \\ \hline
%\hline
%\!\!1.\ {\footnotesize Pay min-wage?}\!\! & 82 & 49 (60\%) & 31 (38\%) &  2~~ (2\%)\\
%\!\!2.\ {\footnotesize Expect quality?}\!\! & 49 & 34 (69\%) &  1~~ (2\%) & 14 (47\%)\\
%\!\!3.\ {\footnotesize Block below?}\!\! & 49 & 10 (20\%) & 16 (33\%) & 23 (47\%)\\
%\begin{table}[h]
%\begin{tabular}{|l|c|c|c|c|}
%\hline
%  & \multicolumn{1}{l|}{Survey}          & \multicolumn{3}{c|}{\%Opinions} \\ \hline
%  &                                      & Good     & Neutral     & Bad    \\ \hline
%1 & Qualtrics Survey                     & 33       & 17          & 50     \\ \hline
%2 & AMT unpaid Survey                    & 60       & 2           & 38    \\ \hline
%\end{tabular}
%\caption{Worker Survey Results}
%\label{tab:table2}
%\end{table}

%Combined overall results are summarized in Figure \ref{fig:WorkerRes}. These include all the surveys mentioned in Table \ref{tab:table2}. As shown in the figure, a total of 39\% participants said it is a bad policy, most of them didn't like the idea of minimum wage for contract workers. 58\% voted in favor of the policy while a small percentage of participants (3\%) believed whether to pay minimum wage or not really depends on the task and whether workers abide by the rules set up by the requesters.
%\begin{figure}[!h]
%\centering
%\ifpdf
%\includegraphics[width=0.9\columnwidth]{ReqRes}
%\fi
%\caption{Summary of requester survey results}
%\label{fig:ReqRes}
%\end{figure}


%\section{Discussion} \label{disc}

%. These patterns are most important and helpful to researchers to make an informed decision in the future in order to make MTurk a better place both for requesters and workers. This paper also contributes to requeuster perspective and projects that more than half of the requeusters are willing to pay the minimum wage to ensure quality.

%RAs seen in Figure \ref{fig:UUSINDworker}, majority of workers in United States are clearly against this policy, as most of them are working on these tasks as their part time income, or they enjoy working on such tasks. Also they are more informed about the tasks, e.g. if a HIT promises them to pay \$0.50 and the amount of time required for it is 30 minutes, then clearly in one hour they would earn only \$1.00. The majority of responses we got in The Unites States affirmed that they are willing to make such informed decisions and would not want to be blocked by anyone from working on such a task. Also some of the responses from disabled workers pointed out that working on crowdsourcing tasks makes them feel like contributing to self well-being, and minimum wage is not really a concern. This has higher implications because in United States, federal minimum wage is \$7.25 per hour, and opinions are more inclined towards no enforcement of the policy. However, the responses we got from India are almost completely in favor of the policy. The minimum wage in India is only \$0.28 per hour, and we would expect at least some of the existing HITs to already meet that requirement, yet there is a stronger urge in the responses to make sure that they are paid minimum wage as most of them depend on MTurk for their basic needs. These trends are almost exact mirror images of each other. This also shows that the two main populations of MTurk which contribute to more than 90\% of tasks on Mturk have completely different beliefs. In the future, different tasks could be designed in a way to fulfill the needs of the workers based on their location.

%Requester opinions are pretty much n favor of the policy as opposed to being against the policy. Even if half of the existing requester population wants to incorporate this policy in order to seek high quality results, we can have a requeuster population satisfying the need of more than half of the MTurk population. Also even though some of them may not be in favor of the policy, if enforced, these requeusters can still get their work done from the other half of the MTurk population which doesn't worry about minimum wage.  For example if the worker is located in India, requesters can make sure they are paid minimum wage but for United States, they can choose to pay any amount they want and should not even block workers from working on tasks that do not pay minimum wage. Some of the requesters also didn't care about blocking a worker from working on tasks if the worker abides by the rules set up by the requeuster. This also means, requeusters could decide to pay the minimum wage after the task completion based on the quality of the work done.

%This study provides a very important result in the context ethics of crowdsourcing concerning fair wage and we believe more interesting results and facts can come forward as and when researchers experiment with this policy.  

%if they want to incorporate this policy in experiments accommodating their research motive. The API can be useful to the requesters who %want to analyze the impact of incorporating the minimum wage paying scheme. Also this could be used to further their research in order to draw interesting conclusions. 

%\subsection{API usage details}

cscw14_minwage.bbl
\begin{thebibliography}{10}

\bibitem{flsa}
Cherry, M.
\newblock {Working for (virtually) minimum wage: Applying the Fair Labor
  Standards Act in cyberspace.}
\newblock {\em Alabama Law Rev. 60(5)\/} (2009), 1077-- 1110.

\bibitem{dolan}
Dolan, K.~A.
\newblock {Samasource Taps Silicon Valley To Create Jobs For Poor People.
  Forbes}, 2011.

\bibitem{felstiner2010working}
Felstiner, A.~L.
\newblock Working the crowd: employment and labor law in the crowdsourcing
  industry.
\newblock {\em Online\/} (2010).
\newblock \url{works.bepress.com/alek_felstiner/1/}.

\bibitem{heymann2011turkalytics}
Heymann, P., and Garcia-Molina, H.
\newblock {Turkalytics: analytics for human computation}.
\newblock In {\em Proc.\ of the 20th Intl.\ WWW Conference} (2011), 477--486.

\bibitem{horton}
Horton, J.~J., and Chilton, L.~B.
\newblock The labor economics of paid crowdsourcing.
\newblock In {\em Proc.\ of the 11th ACM Conference on Electronic Commerce}
  (2010), 209--218.

\bibitem{demog}
Ipeirotis, P.~G.
\newblock Demographics of mechanical turk.
\newblock In {\em CeDER-10-1 working paper- The Center for Digital Economy
  Research} (2010).

\bibitem{irani}
Irani, L.~C., and Silberman, M.~S.
\newblock Turkopticon: Interrupting worker invisibility in amazon mechanical
  turk.
\newblock In {\em Proc.\ of ACM SIGCHI}, ACM (2013), 611--620.

\bibitem{kaufmann}
Kaufmann, N., Schulze, T., and Veit, D.
\newblock More than fun and money. worker motivation in crowdsourcing - a study
  on mechanical turk.
\newblock In {\em AMCIS}, Association for Information Systems (2011).

\bibitem{Kittur13}
Kittur, A., Nickerson, J.~V., Bernstein, M., Gerber, E., Shaw, A., Zimmerman,
  J., Lease, M., and Horton, J.
\newblock {The Future of Crowd Work}.
\newblock In {\em {Proceedings of the ACM Conference on Computer Supported
  Cooperative Work (CSCW)}} (2013), 1301--1318.

\bibitem{martin}
Martin, D., Hanrahan, B.~V., O'Neill, J., and Gupta, N.
\newblock Being a turker.
\newblock In {\em Proc.\ of the 17th ACM Conference on Computer Supported
  Cooperative Work \& Social Computing}, ACM (2014), 224--235.

\bibitem{mason2012conducting}
Mason, W., and Suri, S.
\newblock {Conducting behavioral research on Amazon's Mechanical Turk}.
\newblock {\em Behavior Research Methods 44}, 1 (2012), 1--23.

\bibitem{Ross:2010:CSD:1753846.1753873}
Ross, J., Irani, L., Silberman, M.~S., Zaldivar, A., and Tomlinson, B.
\newblock Who are the crowdworkers?: Shifting demographics in mechanical turk.
\newblock In {\em Ext. Abstracts CHI 2010}, ACM Press (2010), 2863--2872.

\bibitem{silberman}
Silberman, M.~S., Irani, L., and Ross, J.
\newblock Ethics and tactics of professional crowdwork.
\newblock {\em ACM XRDS 17}, 2 (Dec. 2010), 39--43.

\bibitem{wikipedia}
Wikipedia.
\newblock List of minimum wages by country.
\newblock \url{en.wikipedia.org/wiki/List_of_minimum_wages_by_country}.

\bibitem{worshall}
Worshall, T.
\newblock {On the New York Times' Stupidity over Amazon's mechanical Turk.}
\newblock {\em Forbes\/} (2013).

\end{thebibliography}
