<p>The idea of merging them into one stream just to split them later sounds painful. I've not had a reason to do this myself, but here's where I'd start:</p>

<ul>
<li>When you run the docker container, create a volume such that it will be easy to view/ship logs from the host.</li>
<li>Use something like <a href="https://github.com/papertrail/remote_syslog2" rel="noreferrer">remote_syslog2</a> to ship the logs to your log collector</li>
</ul>

<p>It feels a little less-than-elegant to have to do some setup on the host as well, but if you use something like ansible where you can run a playbook and set that up during your deploy to the box it shouldn't be too bad.</p>

CC BY-SA 4.0
2019-05-27T02:23:53.557Z
This solution can be combined with named pipes, the only problem with named pipes is that they don't work on all systems right now. They don't work on [Mac](https://github.com/docker/for-mac/issues/483)
Jens
CC BY-SA 3.0
2017-03-07T19:57:49.790Z
2017-03-07T23:21:58.750Z
2017-03-07T23:21:58.750Z
bradym
<p>I'm still looking for a merging/splitting approach myself, but meanwhile this approach recommended by Kubernetes documentation seems like a sound solution: <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#using-a-sidecar-container-with-the-logging-agent" rel="noreferrer">Use a sidecar container for each of your separate logs</a>.</p>

<p>A "sidecar" is any docker container that you use alongside another docker container to work with it in some way. In this case for each of your three logs you would have a separate container that scans or tails the logs and outputs to stdout. </p>

<p>This way each of your log-sidecar-containers has its own docker-log from its own stdout. Being separate like this, you can use standard docker (and kubernetes, etc) practices for separating or aggregating. Here's what the Kubernetes page has to say:</p>

<blockquote>
  <p>This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it’s hardly a significant overhead. Additionally, because stdout and stderr are handled by the kubelet, you can use built-in tools like kubectl logs.</p>
</blockquote>

<p>The "separate log streams" stem from the built-in tagging that docker applies to logs from different containers, described in the <a href="https://docs.docker.com/engine/admin/logging/log_tags/" rel="noreferrer">docker documentation here:</a></p>

<blockquote>
  <p>The tag log option specifies how to format a tag that identifies the
  container’s log messages. By default, the system uses the first 12
  characters of the container id. To override this behavior, specify a
  tag option</p>
</blockquote>

CC BY-SA 4.0
2019-10-22T17:16:01.957Z
It is worth mentioning the downside of this log-sidecar-containers approach, quote : "Note, that despite low CPU and memory usage, writing logs to a file and then streaming them to stdout can double disk usage".
I wonder if anyone tries this approach in practice.
yusong
CC BY-SA 3.0
2017-07-28T13:51:02.783Z
2017-08-18T01:38:30.527Z
2017-08-18T01:38:30.527Z
Rhubarb
<p>We have an application that writes three types of logs into three separate files: access logs, generic application logs and system logs. The format (and purpose) of these logs are very different. And we have separate logforwarders that send them separately to our centralised logging system.</p>

<p>Based on the <a href="https://12factor.net/logs" rel="noreferrer">treat logs as event streams</a> principle, we are thinking about moving from using files to stdout. While we know some of the benefit of this approach, this would also mean that we'd get a merged stream of the differently formatted logs, which we would need to split again either before we can send them to our central system (Kibana / Splunk / etc.), or inside there.</p>

<p>We're wondering whether there are any tools or recommendations on how we should approach this situation.</p>

CC BY-SA 3.0
2017-03-07T16:22:36.670Z
I don't think it's worth it. Why work harder to merge and then split log streams, just because of some "principle"? Files are good. Files work. This sounds overengineered. I would say maybe pipe all logs into syslog, with different tags, etc. but I must say, if someone in my team suggested this.. I would be .. disappointed.
Assaf Lavie
CC BY-SA 3.0
2017-03-07T16:24:13.433Z
Because using files have some other types of management nightmares, especially if they are generated from inside a docker container. For now it looks like the disadvantages from switching to stdout outweigh the benefits for our use case, but we are already having trouble with our file based approach
SztupY
CC BY-SA 3.0
2017-03-07T16:28:10.570Z
I don't know about "nightmares". I know that this is they way it's been done for a while now, and there's lots of software out there that helps you do this. Log files are rotated, read with checkpoints - a file is a great abstraction for this. I wouldn't buy into a principle that sells me new nightmares because of fear of old, familiar patterns. Your log records are either written to a file, or handled in memory (at least as long as they're being moved around in the container). Good luck achieving the reliability of log files with in-memory stream splitters & mergers.
Assaf Lavie
CC BY-SA 3.0
2017-03-07T16:35:08.393Z
@AssafLavie You should write it in an answer that can get upvoted. IMHO it's a perfectly valid point of view.
Dan Cornilescu
CC BY-SA 3.0
2017-03-07T16:36:50.387Z
The culture here in this beta is still volatile and unsettled. I'd rather not take on the headache of having to defend something that doesn't directly answer the question from down voters and what not. But thanks.
Assaf Lavie
CC BY-SA 3.0
2017-03-07T16:41:52.737Z
@AssafLavie Fair enough.
Dan Cornilescu
CC BY-SA 3.0
2017-03-07T17:37:21.993Z
What kind of app ? Isn't possible to log directly from app to Elasticsearch ? As it's a matter of sending an http request, all apps should be  able to do it
Tensibai
CC BY-SA 3.0
2017-07-28T09:56:27.100Z
I came here with the same question. The simple fact is that docker's built-in logging functionality depends on everything going to stdout/stderr, hence the logging driver and the ecosystem of 3rd party tools building up around it does too. I'm tempted to dump all my logs in a host volume too, but I know I'm going to have to keep going back and managing that when my containers move to k8s or openshift or gke or whatever, whereas if I follow the docker stdout approach it'll be much smoother. Meanwhile I'll keep looking for an answer to this legitimate question
Rhubarb
<docker><logging>
How to have multiple log streams in docker
SztupY
# How to have multiple log streams in docker

(Asked by: username_0 on 2017-03-07)

We have an application that writes three types of logs into three separate files: access logs, generic application logs and system logs. The format (and purpose) of these logs are very different. And we have separate logforwarders that send them separately to our centralised logging system.

Based on the [treat logs as event streams](https://12factor.net/logs) principle, we are thinking about moving from using files to stdout. While we know some of the benefit of this approach, this would also mean that we'd get a merged stream of the differently formatted logs, which we would need to split again either before we can send them to our central system (Kibana / Splunk / etc.), or inside there.

We're wondering whether there are any tools or recommendations on how we should approach this situation.

**Tags:** docker, logging

> **Comments:**
> **username_1**: I don't think it's worth it. Why work harder to merge and then split log streams, just because of some "principle"? Files are good. Files work. This sounds overengineered. I would say maybe pipe all logs into syslog, with different tags, etc. but I must say, if someone in my team suggested this.. I would be .. disappointed.  
> **username_0**: Because using files have some other types of management nightmares, especially if they are generated from inside a docker container. For now it looks like the disadvantages from switching to stdout outweigh the benefits for our use case, but we are already having trouble with our file based approach  
> **username_1**: I don't know about "nightmares". I know that this is they way it's been done for a while now, and there's lots of software out there that helps you do this. Log files are rotated, read with checkpoints - a file is a great abstraction for this. I wouldn't buy into a principle that sells me new nightmares because of fear of old, familiar patterns. Your log records are either written to a file, or handled in memory (at least as long as they're being moved around in the container). Good luck achieving the reliability of log files with in-memory stream splitters & mergers.  
> **username_2**: @AssafLavie You should write it in an answer that can get upvoted. IMHO it's a perfectly valid point of view.  
> **username_1**: The culture here in this beta is still volatile and unsettled. I'd rather not take on the headache of having to defend something that doesn't directly answer the question from down voters and what not. But thanks.  
> **username_2**: @AssafLavie Fair enough.  
> **username_3**: What kind of app ? Isn't possible to log directly from app to Elasticsearch ? As it's a matter of sending an http request, all apps should be  able to do it  
> **username_4**: I came here with the same question. The simple fact is that docker's built-in logging functionality depends on everything going to stdout/stderr, hence the logging driver and the ecosystem of 3rd party tools building up around it does too. I'm tempted to dump all my logs in a host volume too, but I know I'm going to have to keep going back and managing that when my containers move to k8s or openshift or gke or whatever, whereas if I follow the docker stdout approach it'll be much smoother. Meanwhile I'll keep looking for an answer to this legitimate question  

---

## 2 answers

### Answer by username_4 on 2017-07-28. Score: 16 | Accepted answer

I'm still looking for a merging/splitting approach myself, but meanwhile this approach recommended by Kubernetes documentation seems like a sound solution: [Use a sidecar container for each of your separate logs](https://kubernetes.io/docs/concepts/cluster-administration/logging/#using-a-sidecar-container-with-the-logging-agent).

A "sidecar" is any docker container that you use alongside another docker container to work with it in some way. In this case for each of your three logs you would have a separate container that scans or tails the logs and outputs to stdout.

This way each of your log-sidecar-containers has its own docker-log from its own stdout. Being separate like this, you can use standard docker (and kubernetes, etc) practices for separating or aggregating. Here's what the Kubernetes page has to say:

> This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it’s hardly a significant overhead. Additionally, because stdout and stderr are handled by the kubelet, you can use built-in tools like kubectl logs.

The "separate log streams" stem from the built-in tagging that docker applies to logs from different containers, described in the [docker documentation here:](https://docs.docker.com/engine/admin/logging/log_tags/)

> The tag log option specifies how to format a tag that identifies the
> container’s log messages. By default, the system uses the first 12
> characters of the container id. To override this behavior, specify a
> tag option

> **Comments:**
> **username_7**: It is worth mentioning the downside of this log-sidecar-containers approach, quote : "Note, that despite low CPU and memory usage, writing logs to a file and then streaming them to stdout can double disk usage". I wonder if anyone tries this approach in practice.  

---

### Answer by username_5 on 2017-03-07. Score: 8

The idea of merging them into one stream just to split them later sounds painful. I've not had a reason to do this myself, but here's where I'd start:

* When you run the docker container, create a volume such that it will be easy to view/ship logs from the host.
* Use something like [remote\_syslog2](https://github.com/papertrail/remote_syslog2) to ship the logs to your log collector

It feels a little less-than-elegant to have to do some setup on the host as well, but if you use something like ansible where you can run a playbook and set that up during your deploy to the box it shouldn't be too bad.

> **Comments:**
> **username_6**: This solution can be combined with named pipes, the only problem with named pipes is that they don't work on all systems right now. They don't work on [Mac](https://github.com/docker/for-mac/issues/483)  

---

