  The subjective quality of transmitted speech is traditionally assessed in a
controlled laboratory environment according to ITU-T Rec. P.800. In turn, with
crowdsourcing, crowdworkers participate in a subjective online experiment using
their own listening device, and in their own working environment. Despite such
less controllable conditions, the increased use of crowdsourcing micro-task
platforms for quality assessment tasks has pushed a high demand for
standardized methods, resulting in ITU-T Rec. P.808. This work investigates the
impact of the number of judgments on the reliability and the validity of
quality ratings collected through crowdsourcing-based speech quality
assessments, as an input to ITU-T Rec. P.808 . Three crowdsourcing experiments
on different platforms were conducted to evaluate the overall quality of three
different speech datasets, using the Absolute Category Rating procedure. For
each dataset, the Mean Opinion Scores (MOS) are calculated using differing
numbers of crowdsourcing judgements. Then the results are compared to MOS
values collected in a standard laboratory experiment, to assess the validity of
crowdsourcing approach as a function of number of votes. In addition, the
reliability of the average scores is analyzed by checking inter-rater
reliability, gain in certainty, and the confidence of the MOS. The results
provide a suggestion on the required number of votes per condition, and allow
to model its impact on validity and reliability.
