  In this paper we describe our work on designing a web based, distributed data
analysis system based on the popular MapReduce framework deployed on a small
cloud; developed specifically for analyzing web server logs. The log analysis
system consists of several cluster nodes, it splits the large log files on a
distributed file system and quickly processes them using MapReduce programming
model. The cluster is created using an open source cloud infrastructure, which
allows us to easily expand the computational power by adding new nodes. This
gives us the ability to automatically resize the cluster according to the data
analysis requirements. We implemented MapReduce programs for basic log analysis
needs like frequency analysis, error detection, busy hour detection etc. as
well as more complex analyses which require running several jobs. The system
can automatically identify and analyze several web server log types such as
Apache, IIS, Squid etc. We use open source projects for creating the cloud
infrastructure and running MapReduce jobs.
