<p>There are two ways to accomplish this, though one of them was only available recently.</p>

<p>The old-school version, the one you can do as far back as Logstash 1.5, is to pay attention to tags and use conditionals to separate your inputs. Roughly...</p>

<pre><code>input {
  tcp {
    port =&gt; 1525
    codec =&gt; json_lines
    tags =&gt; [ 'tcp' ]
  }
}

input {
  file {
    path =&gt; '/var/log/app.log'
    codec =&gt; 'json'
    tags =&gt; [ 'file' ]
  }
}

output {
  if 'file' in [tags] {
    elasticsearch {
      host =&gt; 'logstash-es'
      index =&gt; 'files'
    }
  }
  if 'tcp' in [tags] {
    elasticsearch {
      host =&gt; 'logstash-es'
      index =&gt; 'tcp'
    }
  }
}
</code></pre>

<p>This results in two inputs that output to two separate outputs. This is all one file, though. Elastic figured out people were muxing pipelines this way, and came up with <a href="https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html" rel="nofollow noreferrer">a way to do multiple pipelines in separate files</a>.</p>

<pre><code>- pipeline.id: tcp-inputs
  path.config: '/etc/logstash/pipelines/tcp.cfg'
  pipeline.workers: 3
- pipeline.id: file-inputs
  path.config: '/etc/logstash/pipelines/files.cfg'
  pipeline.workers: 2
</code></pre>

<p>This approach is somewhat more maintainable since the pipelines are in separate files, and humans don't have to reason out how the flows work when presented in a single big file. Pipelines are available in Logstash 6.0 and newer.</p>

CC BY-SA 4.0
2018-08-14T12:59:59.787Z
2018-08-14T12:59:59.787Z
sysadmin1138
<p>Once parsed your config create one and only one pipeline, with various inputs, various filters and various outputs. </p>

<p>You have to use some conditional constructs to apply filter and output only to specific messages, usually using a special tag or field set on input.</p>

<p>Some exemple are available in <a href="https://www.elastic.co/guide/en/logstash/current/config-examples.html#using-conditionals" rel="nofollow noreferrer">logstash documentation</a>:</p>

<blockquote>
<pre><code>output {
  if [type] == "apache" {
    if [status] =~ /^5\d\d/ {
      nagios { ...  }
    } else if [status] =~ /^4\d\d/ {
      elasticsearch { ... }
    }
    statsd { increment =&gt; "apache.%{status}" }
  }
}
</code></pre>
</blockquote>

CC BY-SA 3.0
2018-04-16T09:19:33.923Z
I tried it with conditionals and had the same outcome
Saar Grin
CC BY-SA 3.0
2018-04-16T09:20:09.650Z
Turns out the proper way to do it is with pipelines.yml file which i somehow missed.
Saar Grin
CC BY-SA 3.0
2018-04-16T09:30:33.983Z
Concerning your first comment, without what you tried, there's no way to tell what could have been wrong.
Tensibai
CC BY-SA 3.0
2018-04-16T09:14:12.773Z
2018-04-16T09:14:12.773Z
Tensibai
<p>I'm trying to set up a centralized syslog for multiple log sources.</p>

<p>So I have a logstash that has two separate inputs and two separate outputs</p>

<p>However for some reason the data from one of the inputs ends up in both indexes</p>

<p>What am i doing wrong?</p>

<p>Below are both pipelines' configs</p>

<pre><code>input {
  tcp {
    port =&gt; 5052
    codec =&gt; "json_lines"
  }
}
output {
  elasticsearch {
    hosts =&gt; "10.50.6.116"
    index =&gt; "remote"
  }
  file {
    path =&gt; "/var/log/logstash/remote-tcp.log"
  }
  stdout { codec =&gt; rubydebug }
}

input {
  file {
    path =&gt; "/data/vmlist/*.csv"
    start_position =&gt; "beginning"
    sincedb_path =&gt; "/tmp/sincedb"
  }
}
filter {
  csv {
    separator =&gt; ","
    columns =&gt; ["VM Name","Creation Date","Owner","Type","Message"]
  }
}
output {
  elasticsearch {
    hosts =&gt; "http://10.50.6.116:9200"
    index =&gt; "vms"
    document_type =&gt; "csv"
  }
  stdout{ codec=&gt; rubydebug}
}
</code></pre>

<elastic><logstash>
Logstash multiple pipelines going into same index
Saar Grin
# Logstash multiple pipelines going into same index

(Asked by: username_0 on 2018-04-15)

I'm trying to set up a centralized syslog for multiple log sources.
So I have a logstash that has two separate inputs and two separate outputs
However for some reason the data from one of the inputs ends up in both indexes
What am i doing wrong?
Below are both pipelines' configs
input {
  tcp {
    port => 5052
    codec => "json_lines"
  }
}
output {
  elasticsearch {
    hosts => "10.50.6.116"
    index => "remote"
  }
  file {
    path => "/var/log/logstash/remote-tcp.log"
  }
  stdout { codec => rubydebug }
}

input {
  file {
    path => "/data/vmlist/*.csv"
    start_position => "beginning"
    sincedb_path => "/tmp/sincedb"
  }
}
filter {
  csv {
    separator => ","
    columns => ["VM Name","Creation Date","Owner","Type","Message"]
  }
}
output {
  elasticsearch {
    hosts => "http://10.50.6.116:9200"
    index => "vms"
    document_type => "csv"
  }
  stdout{ codec=> rubydebug}
}

**Tags:** elastic, logstash


---

## 2 answers

### Answer by username_1 on 2018-08-14. Score: 2

There are two ways to accomplish this, though one of them was only available recently.
The old-school version, the one you can do as far back as Logstash 1.5, is to pay attention to tags and use conditionals to separate your inputs. Roughly...
input {
  tcp {
    port => 1525
    codec => json_lines
    tags => [ 'tcp' ]
  }
}

input {
  file {
    path => '/var/log/app.log'
    codec => 'json'
    tags => [ 'file' ]
  }
}

output {
  if 'file' in [tags] {
    elasticsearch {
      host => 'logstash-es'
      index => 'files'
    }
  }
  if 'tcp' in [tags] {
    elasticsearch {
      host => 'logstash-es'
      index => 'tcp'
    }
  }
}

This results in two inputs that output to two separate outputs. This is all one file, though. Elastic figured out people were muxing pipelines this way, and came up with a way to do multiple pipelines in separate files.
- pipeline.id: tcp-inputs
  path.config: '/etc/logstash/pipelines/tcp.cfg'
  pipeline.workers: 3
- pipeline.id: file-inputs
  path.config: '/etc/logstash/pipelines/files.cfg'
  pipeline.workers: 2

This approach is somewhat more maintainable since the pipelines are in separate files, and humans don't have to reason out how the flows work when presented in a single big file. Pipelines are available in Logstash 6.0 and newer.


---

### Answer by username_2 on 2018-04-16. Score: 0

Once parsed your config create one and only one pipeline, with various inputs, various filters and various outputs. 
You have to use some conditional constructs to apply filter and output only to specific messages, usually using a special tag or field set on input.
Some exemple are available in logstash documentation:

output {
  if [type] == "apache" {
    if [status] =~ /^5\d\d/ {
      nagios { ...  }
    } else if [status] =~ /^4\d\d/ {
      elasticsearch { ... }
    }
    statsd { increment => "apache.%{status}" }
  }
}

> **Comments:**
> **username_0**: I tried it with conditionals and had the same outcome  
> **username_0**: Turns out the proper way to do it is with pipelines.yml file which i somehow missed.  
> **username_2**: Concerning your first comment, without what you tried, there's no way to tell what could have been wrong.  

---

