  Most general population web surveys are based on online panels maintained by
commercial survey agencies. However, survey agencies differ in their panel
selection and management strategies. Little is known if these different
strategies cause differences in survey estimates. This paper presents the
results of a systematic study designed to analyze the differences in web survey
results between agencies. Six different survey agencies were commissioned with
the same web survey using an identical standardized questionnaire covering
factual health items. Five surveys were fielded at the same time. A calibration
approach was used to control the effect of demographics on the outcome.
Overall, the results show differences between probability and non-probability
surveys in health estimates, which were reduced but not eliminated by
weighting. Furthermore, the differences between non-probability surveys before
and after weighting are larger than expected between random samples from the
same population.
