  A typical web search engine consists of three principal parts: crawling
engine, indexing engine, and searching engine. The present work aims to
optimize the performance of the crawling engine. The crawling engine finds new
web pages and updates web pages existing in the database of the web search
engine. The crawling engine has several robots collecting information from the
Internet. We first calculate various performance measures of the system (e.g.,
probability of arbitrary page loss due to the buffer overflow, probability of
starvation of the system, the average time waiting in the buffer). Intuitively,
we would like to avoid system starvation and at the same time to minimize the
information loss. We formulate the problem as a multi-criteria optimization
problem and attributing a weight to each criterion. We solve it in the class of
threshold policies. We consider a very general web page arrival process modeled
by Batch Marked Markov Arrival Process and a very general service time modeled
by Phase-type distribution. The model has been applied to the performance
evaluation and optimization of the crawler designed by INRIA Maestro team in
the framework of the RIAM INRIA-Canon research project.
