These are some handy scripts to launch / monitor / manage Amazon EMR jobs.
We use these tools to launch and monitor EMR jobs.
As we have seen we can supply any number of arguments in command line. But when doing so, we are also hardcoding the parameter's positional values. First parameter is input_dir, second parameter is output_dir …etc. This can lead to inflexible programs. What if we want to over write only the third argument? We still have to supply all the args. And if we need to pass in a lot of arguments, then this method is not very handy.
Lets put all our arguments into a Property file and feed that to our program. Java property files are plain text files containing key=value per line.
Here the properties are prefixed with 'my' just so we don't end up override any system properties by accident.
Next we should place this property file in HDFS or S3.
it can be used for launching MR jobs.
There is a Web UI to submit a MapReduce job and monitor its progress. We will look at an alternative – launching an EMR job from command line and monitoring its progress.
The following scripts do this.
Launch script is split in two parts. First part is configurable. Second part of the script is generic and does not need to be changed all much. That is why I have split the script this way. The bottom script can be 'called' from any script.
larger job needs 20 instances we can get 20.
called from our run script.
We had launched a EMR job with default configuration, how about if we want to tweak hadoop settings? The following script shows how to do that.
Also, all these logs will go away after our cluster terminates.
Also MRJob framework, developed by Yelp engineers and open sourced is also worth a look. It is a Python based framework.
This article is a great one. It describes all clearly. I really liked it. It's very helpful. I was looking for such articles. I have read a article here "http://www.techyv.com/article/amazon-elastic-mapreduce-cloud-computing" which is almost like this.