  In this paper we review studies of the growth of the Internet and
technologies that are useful for information search and retrieval on the Web.
Search engines are retrieve the efficient information. We collected data on the
Internet from several different sources, e.g., current as well as projected
number of users, hosts, and Web sites. The trends cited by the sources are
consistent and point to exponential growth in the past and in the coming
decade. Hence it is not surprising that about 85% of Internet users surveyed
claim using search engines and search services to find specific information and
users are not satisfied with the performance of the current generation of
search engines; the slow retrieval speed, communication delays, and poor
quality of retrieved results. Web agents, programs acting autonomously on some
task, are already present in the form of spiders, crawler, and robots. Agents
offer substantial benefits and hazards, and because of this, their development
must involve attention to technical details. This paper illustrates the
different types of agents,crawlers, robots,etc for mining the contents of web
in a methodical, automated manner, also discusses the use of crawler to gather
specific types of information from Web pages, such as harvesting e-mail
addresses
