  A publicly available dataset for federated search reflecting a real web
environment has long been absent, making it difficult for researchers to test
the validity of their federated search algorithms for the web setting. We
present several experiments and analyses on resource selection on the web using
a recently released test collection containing the results from more than a
hundred real search engines, ranging from large general web search engines such
as Google, Bing and Yahoo to small domain-specific engines. First, we
experiment with estimating the size of uncooperative search engines on the web
using query based sampling and propose a new method using the ClueWeb09
dataset. We find the size estimates to be highly effective in resource
selection. Second, we show that an optimized federated search system based on
smaller web search engines can be an alternative to a system using large web
search engines. Third, we provide an empirical comparison of several popular
resource selection methods and find that these methods are not readily suitable
for resource selection on the web. Challenges include the sparse resource
descriptions and extremely skewed sizes of collections.
