  The laborious and costly nature of affect annotation is a key detrimental
factor for obtaining large scale corpora with valid and reliable affect labels.
Motivated by the lack of tools that can effectively determine an annotator's
reliability, this paper proposes general quality assurance (QA) tests for
real-time continuous annotation tasks. Assuming that the annotation tasks rely
on stimuli with audiovisual components, such as videos, we propose and evaluate
two QA tests: a visual and an auditory QA test. We validate the QA tool across
20 annotators that are asked to go through the test followed by a lengthy task
of annotating the engagement of gameplay videos. Our findings suggest that the
proposed QA tool reveals, unsurprisingly, that trained annotators are more
reliable than the best of untrained crowdworkers we could employ. Importantly,
the QA tool introduced can predict effectively the reliability of an affect
annotator with 80% accuracy, thereby, saving on resources, effort and cost, and
maximizing the reliability of labels solicited in affective corpora. The
introduced QA tool is available and accessible through the PAGAN annotation
platform.
