  Question-answer driven Semantic Role Labeling (QA-SRL) was proposed as an
attractive open and natural flavour of SRL, potentially attainable from laymen.
Recently, a large-scale crowdsourced QA-SRL corpus and a trained parser were
released. Trying to replicate the QA-SRL annotation for new texts, we found
that the resulting annotations were lacking in quality, particularly in
coverage, making them insufficient for further research and evaluation. In this
paper, we present an improved crowdsourcing protocol for complex semantic
annotation, involving worker selection and training, and a data consolidation
phase. Applying this protocol to QA-SRL yielded high-quality annotation with
drastically higher coverage, producing a new gold evaluation dataset. We
believe that our annotation protocol and gold standard will facilitate future
replicable research of natural semantic annotations.
