  Nowadays, more and more people use the Web as their primary source of
up-to-date information. In this context, fast crawling and indexing of newly
created Web pages has become crucial for search engines, especially because
user traffic to a significant fraction of these new pages (like news, blog and
forum posts) grows really quickly right after they appear, but lasts only for
several days.
  In this paper, we study the problem of timely finding and crawling of such
ephemeral new pages (in terms of user interest). Traditional crawling policies
do not give any particular priority to such pages and may thus crawl them not
quickly enough, and even crawl already obsolete content. We thus propose a new
metric, well thought out for this task, which takes into account the decrease
of user interest for ephemeral pages over time.
  We show that most ephemeral new pages can be found at a relatively small set
of content sources and present a procedure for finding such a set. Our idea is
to periodically recrawl content sources and crawl newly created pages linked
from them, focusing on high-quality (in terms of user interest) content. One of
the main difficulties here is to divide resources between these two activities
in an efficient way. We find the adaptive balance between crawls and recrawls
by maximizing the proposed metric. Further, we incorporate search engine click
logs to give our crawler an insight about the current user demands. Efficiency
of our approach is finally demonstrated experimentally on real-world data.
