<NME> README.md
<BEF> # logstash-kafka

### Status
[![Build Status](https://travis-ci.org/joekiller/logstash-kafka.svg?branch=master)](https://travis-ci.org/joekiller/logstash-kafka)

This project implements Kafka 0.8.2.1 inputs and outputs for logstash 1.4.X only. For 1.5 support read below.

For more info about logstash, see <http://logstash.net/>

logstash-kafka has been intergrated into [logstash-input-kafka](https://github.com/logstash-plugins/logstash-input-kafka) and [logstash-output-kafka](https://github.com/logstash-plugins/logstash-output-kafka).  
It will be released with the 1.5 version of logstash.  Thank you for your support.  My goal is to close out tickets here 
but for the most part, issues and problems should be resolved via:

  * https://groups.google.com/forum/#!forum/logstash-users The logstash-users mailing list is extremely active and I 
  participate especially in Kafka troubleshooting/discussion there.
  
  * IRC: #logstash on irc.freenode.org
  
  * If you really have an issue that you can't resolve via the mailing list or IRC channel, try posting an issue to
  github.  Please make sure that issues are actually software issues, not configuration issues as those should be left
  to the mailing lists and IRC. https://github.com/elasticsearch/logstash/issues

I will keep this library up to date at least until logstash 1.5 is released.

I will continue helping with the plugin and readying for the next version of Kafka with the jruby-kafka library.

## Dependencies

* [Apache Kafka] version 0.8.2.1 

* [jruby-kafka] library.

[Apache Kafka]: http://kafka.apache.org/
Now install dependencies:

    # Install logstash ruby dependencies
    ruby gembag.rb logstash.gemspec

    # to use logstash gems or libraries in irb, use the following
    # this gets you an 'irb' shell with logstash's environment
To simply build the logstash jar as is with Kafka enabled run:

    # make tarball

## Building Logstash-Kafka RPM

**Note that this doesn't build a logstash RPM but an RPM that will install the logstash-kafka libraries on top of an existing logstash installation**

To build an rpm

    # make package

Installing the resulting rpm after installing logstash from the elasticsearch repo will copy the kafka plugin and dependencies into `/opt/logstash`.


## Building Logstash-Kafka Gem

You can build the gem file and then install it to logstash.

    # gem build logstash-kafka.gemspec
    Finished in 0.123 seconds
    19 examples, 0 failures

Alternately, if you have just built the flatjar or jar, you can run the tests
specifically on those like so:

    % make flatjar-test
    # or ...
    % make jar-test

Finally, like 'bin/logstash rspec' above, you can invoke the jar to run a test
like so:

    % java -jar logstash.jar rspec spec/filters/grok.rb
    ...................
```
 

### Manual Logstash-Kafka Gem Install on Logstash 1.4.2

As root or using sudo:

```bash
cd /path/to/logstash

    make flatjar

rpm, deb, or other package formats are not currently available, but are easy to
build with [fpm](https://github.com/jordansissel/fpm). If you are interested in
seeing future releases include your favorite packaging format, please let me
know.

## Project Principles


## Running

You should run this version of logstash the same as you would the normal logstash with:

    # bin/logstash -f logstash.conf

Contributed plugins can also still be installed for Logstash 1.4+ using:

    # bin/plugin install contrib

### Input

See http://kafka.apache.org/documentation.html#consumerconfigs for details about the Kafka consumer options.

    # input {
        kafka {
            zk_connect => ... # string (optional), default: "localhost:2181"
            group_id => ... # string (optional), default: "logstash"
            topic_id => ... # string (optional), default: "test"
            reset_beginning => ... # boolean (optional), default: false
            consumer_threads => ... # number (optional), default: 1
            queue_size => ... # number (optional), default: 20
            rebalance_max_retries => ... # number (optional), default: 4
            rebalance_backoff_ms => ... # number (optional), default:  2000
            consumer_timeout_ms => ... # number (optional), default: -1
            consumer_restart_on_error => ... # boolean (optional), default: true
            consumer_restart_sleep_ms => ... # number (optional), default: 0
            decorate_events => ... # boolean (optional), default: false
            consumer_id => ... # string (optional) default: nil
            fetch_message_max_bytes => ... # number (optional) default: 1048576
        }
    }

### Output

See http://kafka.apache.org/documentation.html#producerconfigs for details about the Kafka producer options.

    # output {
        kafka {
            broker_list => ... # string (optional), default: "localhost:9092"
            topic_id => ... # string (optional), default: "test"
            compression_codec => ... # string (optional), one of ["none", "gzip", "snappy"], default: "none"
            compressed_topics => ... # string (optional), default: ""
            request_required_acks => ... # number (optional), one of [-1, 0, 1], default: 0
            serializer_class => ... # string, (optional) default: "kafka.serializer.StringEncoder"
            partitioner_class => ... # string (optional) default: "kafka.producer.DefaultPartitioner"
            request_timeout_ms => ... # number (optional) default: 10000
            producer_type => ... # string (optional), one of ["sync", "async"] default => 'sync'
            key_serializer_class => ... # string (optional) default: "kafka.serializer.StringEncoder"
            message_send_max_retries => ... # number (optional) default: 3
            retry_backoff_ms => ... # number (optional) default: 100
            topic_metadata_refresh_interval_ms => ... # number (optional) default: 600 * 1000
            queue_buffering_max_ms => ... # number (optional) default: 5000
            queue_buffering_max_messages => ... # number (optional) default: 10000
            queue_enqueue_timeout_ms => ... # number (optional) default: -1
            batch_num_messages => ... # number (optional) default: 200
            send_buffer_bytes => ... # number (optional) default: 100 * 1024
            client_id => ... # string (optional) default: ""
        }
    }

The default codec is json for input and outputs.  If you select a codec of plain, logstash will encode your messages with not only the message
but also with a timestamp and hostname.  If you do not want anything but your message passing through, you should make
the output configuration something like:

    # output {
        kafka {
            codec => plain {
                format => "%{message}"
            }
        }
    }

## Testing

    # make tarball
    # make tarball-test

## Notes

The make file is updated to work with Logstash 1.4.0+.  DEB package building isn't supported at this time.

<MSG> Update docs on dev/usage

<DFF> @@ -34,7 +34,9 @@ Otherwise, here's how to get started with rvm:
 Now install dependencies:
 
     # Install logstash ruby dependencies
-    ruby gembag.rb logstash.gemspec
+    bin/logstash deps
+
+Other commands:
 
     # to use logstash gems or libraries in irb, use the following
     # this gets you an 'irb' shell with logstash's environment
@@ -61,15 +63,17 @@ rspec <some spec>` will suffice:
     Finished in 0.123 seconds
     19 examples, 0 failures
 
-Alternately, if you have just built the flatjar or jar, you can run the tests
+Alternately, if you have just built the flatjar, you can run the tests
 specifically on those like so:
 
-    % make flatjar-test
-    # or ...
-    % make jar-test
+    make flatjar-test
+
+If you want to run all the tests from source (not compiled jar), do:
 
-Finally, like 'bin/logstash rspec' above, you can invoke the jar to run a test
-like so:
+    make test
+
+Finally, like 'bin/logstash rspec' above, you can invoke the jar to run a
+specific test like so:
 
     % java -jar logstash.jar rspec spec/filters/grok.rb
     ...................
@@ -85,10 +89,9 @@ If you want to build the jar yourself, run:
 
     make flatjar
 
-rpm, deb, or other package formats are not currently available, but are easy to
-build with [fpm](https://github.com/jordansissel/fpm). If you are interested in
-seeing future releases include your favorite packaging format, please let me
-know.
+You can build rpms and debs, if you need those. Building rpms requires you have [fpm](github.com/jordansissel/fpm), then do this:
+
+    make package
 
 ## Project Principles
 

Update docs on dev/usage