Margin of error
If this is all based on chance, how can we be sure that these numbers will always be correct? The answer is, we cannot. If random samples of 400 are selected from a universe, one after another, eventually we could expect one of them to turn up a result that falls outside the 5% margin of error. And if a hundred different random samples of 400 were selected, on average, five of them would fall beyond the predicted 5% margin of error. Which sample that might be we have no way of knowing. It might be the first and only sample we draw; it might not come until sample number 17 or sample number 26.
This means that no matter how carefully the survey is constructed and executed there is a chance, albeit just one chance in 20, that the results will be quite incorrect.
Put more positively, this means that the chances are 95 out of 100 that the results will indeed fall within the calculated margin of error. For most practical purposes this 95% level of confidence, as it is called, constitutes quite favorable odds. It is not a sure bet, but it is a rather safe bet. The level of confidence and margin of error are mathematically related. Each is a trade-off against the other. If the level of confidence is higher, the margin of error is greater. To make the margin of error smaller, the level of confidence must go down.
In the case of a sample of 400, if the level of confidence is raised from 95% to 99.6% the margin of error goes from 5% to 7.5%. If we wanted to go in the other direction with the margin of error, for a sample of 400, reducing margin of error to 2.5% would drop the level of confidence to 68%.
In other words, if the pollster with a random sample of 400 wants to be able to claim to be within 2.5%, he will have to admit that the chances of having a bad sample are 68%, or about one in three. On the other hand, he can say he is 99.6% sure that his percentages are correct, but that would be within a ±7.5% margin of error.
The 99.6% certainty sounds good, but a 15-percentage-point range is rather broad. The narrower margin of error, 2.5%, is attractive, but having to confess to a one-in-three chance that the numbers are wrong is not appealing to most polling organizations. As a result, virtually all polling organizations take the middle road and opt for a 95% level of confidence.
As for the margin of error, most national polls want to get closer than 5%. The way to reduce the margin of error is to increase the sample size. Unfortunately, the margin of error goes down rather slowly as sample size increases. A sample of 1,000 has a margin of error of 4%. At 1,500 it drops to 3%. To get to 2% the sample would need to be about 4,000.
Here too, polling organizations are faced with a trade-off. A smaller margin of error would be desirable, but it is expensive. In light of the vastly increased cost of conducting 4,000 as opposed to 1,500 interviews, compared to the difference in the margin of error, 3% rather than 2%, most national polls settle for a sample somewhere between 1,500 and 2,000.
One reason for a somewhat larger sample size is to permit over sampling of one or more subgroups of the population. A sample of 1,500 has a margin of error of 3%. However, a subgroup of that sample will have a larger margin of error. The margin of error for all women in the sample may be 4%, and for all married women 5%.
Some polls report the margin of error for each of the subgroups which they mention separately, others do not. In either case, the mathematical fact remains: a smaller sample size means a larger margin of error.
Provisos about these sundry parameters usually appear at the end of the article or report with the polling data. The sample size and dates of interviewing are given, and something like the language of the Merit Report is included, "Based on a sample of this size, one can say with 95% certainty that the accuracy of the results is within plus or minus three percentage points of what the results would have been had the entire adult population living in telephone households been polled." While that certainly sums up the points made thus far in this section, one more caveat is in order.