  Computational models of argument quality (AQ) have focused primarily on
assessing the overall quality or just one specific characteristic of an
argument, such as its convincingness or its clarity. However, previous work has
claimed that assessment based on theoretical dimensions of argumentation could
benefit writers, but developing such models has been limited by the lack of
annotated data. In this work, we describe GAQCorpus, the first large,
domain-diverse annotated corpus of theory-based AQ. We discuss how we designed
the annotation task to reliably collect a large number of judgments with
crowdsourcing, formulating theory-based guidelines that helped make subjective
judgments of AQ more objective. We demonstrate how to identify arguments and
adapt the annotation task for three diverse domains. Our work will inform
research on theory-based argumentation annotation and enable the creation of
more diverse corpora to support computational AQ assessment.
