  Users of web search engines are known to mostly focus on the top ranked
results of the search engine result page. While many studies support this well
known information seeking pattern only few studies concentrate on the question
what users are missing by neglecting lower ranked results. To learn more about
the relevance distributions in the so-called long tail we conducted a relevance
assessment study with the Million Short long-tail web search engine. While we
see a clear difference in the content between the head and the tail of the
search engine result list we see no statistical significant differences in the
binary relevance judgments and weak significant differences when using graded
relevance. The tail contains different but still valuable results. We argue
that the long tail can be a rich source for the diversification of web search
engine result lists but it needs more evaluation to clearly describe the
differences.
