  In image quality assessment, a collective visual quality score for an image
or video is obtained from the individual ratings of many subjects. One commonly
used format for these experiments is the two-alternative forced choice method.
Two stimuli with the same content but differing visual quality are presented
sequentially or side-by-side. Subjects are asked to select the one of better
quality, and when uncertain, they are required to guess. The relaxed
alternative forced choice format aims to reduce the cognitive load and the
noise in the responses due to the guessing by providing a third response
option, namely, ``not sure''. This work presents a large and comprehensive
crowdsourcing experiment to compare these two response formats: the one with
the ``not sure'' option and the one without it. To provide unambiguous ground
truth for quality evaluation, subjects were shown pairs of images with
differing numbers of dots and asked each time to choose the one with more dots.
Our crowdsourcing study involved 254 participants and was conducted using a
within-subject design. Each participant was asked to respond to 40 pair
comparisons with and without the ``not sure'' response option and completed a
questionnaire to evaluate their cognitive load for each testing condition. The
experimental results show that the inclusion of the ``not sure'' response
option in the forced choice method reduced mental load and led to models with
better data fit and correspondence to ground truth. We also tested for the
equivalence of the models and found that they were different. The dataset is
available at http://database.mmsp-kn.de/cogvqa-database.html.
