  We investigate the feasibility of obtaining highly trustworthy results using
crowdsourcing on complex engineering tasks. Crowdsourcing is increasingly seen
as a potentially powerful way of increasing the supply of labor for solving
society's problems. While applications in domains such as citizen-science,
citizen-journalism or knowledge organization (e.g., Wikipedia) have seen many
successful applications, there have been fewer applications focused on solving
engineering problems, especially those involving complex tasks. This may be in
part because of concerns that low quality input into engineering analysis and
design could result in failed structures leading to loss of life. We compared
the quality of work of the anonymous workers of Amazon Mechanical Turk (AMT),
an online crowdsourcing service, with the quality of work of expert engineers
in solving the complex engineering task of evaluating virtual wind tunnel data
graphs. On this representative complex engineering task, our results showed
that there was little difference between expert engineers and crowdworkers in
the quality of their work and explained reasons for these results. Along with
showing that crowdworkers are effective at completing new complex tasks our
paper supplies a number of important lessons that were learned in the process
of collecting this data from AMT, which may be of value to other researchers.
