  The vastness of the web imposes a prohibitive cost on building large-scale
search engines with limited resources. Crawl frontiers thus need to be
optimized to improve the coverage and freshness of crawled content. In this
paper, we propose an approach for modeling the dynamics of change in the web
using archived copies of webpages. To evaluate its utility, we conduct a
preliminary study on the scholarly web using 19,977 seed URLs of authors'
homepages obtained from their Google Scholar profiles. We first obtain archived
copies of these webpages from the Internet Archive (IA), and estimate when
their actual updates occurred. Next, we apply maximum likelihood to estimate
their mean update frequency ($\lambda$) values. Our evaluation shows that
$\lambda$ values derived from a short history of archived data provide a good
estimate for the true update frequency in the short-term, and that our method
provides better estimations of updates at a fraction of resources compared to
the baseline models. Based on this, we demonstrate the utility of archived data
to optimize the crawling strategy of web crawlers, and uncover important
challenges that inspire future research directions.
