The amount of information on the World Wide Web is outstripping the ability of search engines to index it, according to a new study.
There are about 800 million publicly available pages on the Web, but even the best search engines index only about 16 per cent of them. Two years ago, the best engines indexed a third of the 320 million pages that were on the Web (This Week, 28 June 1997, p 12).
Steve Lawrence and C. Lee Giles of the NEC Research Institute in Princeton, New Jersey, arrived at their estimate of the size of the Web by sampling all possible Internet addresses to see which ones actually represented servers, then sampling servers to see how much information was available. They found 2.8 million servers with an average of 289 publicly available pages each.
But when they submitted test queries to the 11 most popular search ...
To continue reading this article, subscribe to receive access to all of newscientist.com, including 20 years of archive content.