<h2>Graylog</h2>
<p>Since two people already advised you to rethink your current process (which I second since it will cause you sleepless nights at some point ;)), I will go another route and recommend a specific piece of software which - in my opinion - fits most of your needs: <a href="https://www.graylog.org/" rel="nofollow noreferrer">Graylog</a>.</p>
<p>I implemented and used a couple of <a href="https://www.elastic.co/webinars/introduction-elk-stack" rel="nofollow noreferrer">ELK stacks</a> for both log aggregation and business intelligence and also run/maintain graylog for about two years now at my current employer. I recommend graylog since it has the following features built-in and is - in my opinion - a bit easier to setup and maintain:</p>
<ul>
<li>A web-interface</li>
<li>Multiuser capabilities</li>
<li>Alerting</li>
</ul>
<p>As far as I understand your scenario, it looks as if you need to act or be alerted on certain events that appear in your stream of log messages. If we look at the <a href="https://www.graylog.org/features" rel="nofollow noreferrer">Graylog features</a>:</p>
<blockquote>
<p>Trigger actions or get notified when something needs attention, such as failed login attempts, exceptions or performance degradation.</p>
<p>Ideas: Send an email or Slack message to your team. Spawn a new machine to balance the processing load. Block IP ranges in your firewalls automatically when an attack is detected.</p>
</blockquote>
<p>To give graylog a try I'd recommend the following two steps:</p>
<ul>
<li>Setup a dedicated host which is reachable by all you application hosts to run graylog (and its dependencies MongoDB and ElasticSearch)</li>
<li>Send logs from your application to graylog (possibly as <a href="http://docs.graylog.org/en/2.3/pages/gelf.html" rel="nofollow noreferrer">GELF</a> messages)</li>
</ul>
<p>Note: These two steps have the ability to fill pages and pages of best practices and should receive at least a couple of thoughts. Not to mention that graylog isn't a monitoring solution and graylog itself should be monitored by a proper monitoring tool (like e.g. Icinga, Prometheus, Nagios to name just a few).</p>

CC BY-SA 3.0
2017-10-25T07:02:49.860Z
2017-10-25T08:20:23.313Z
2020-06-18T08:28:10.803Z
malte
<p>I agree with @Xiong Chiamiov and I want to give more clarify option.
If you want every line in the CLI to be monitored, I would suggest to redirect all the output to specific file and the error to another file, then use <a href="https://www.elastic.co/products/logstash" rel="nofollow noreferrer">logstash</a> or <a href="https://www.elastic.co/products/beats/filebeat" rel="nofollow noreferrer">filebeat</a> to send both of this files to <a href="https://www.elastic.co/products/elasticsearch" rel="nofollow noreferrer">Elasticsearch</a>, then you can configure <a href="https://github.com/sivasamyk/logtrail" rel="nofollow noreferrer">Logtril</a> with <a href="https://www.elastic.co/products/kibana" rel="nofollow noreferrer">Kibana</a> to give you view, analyze, search and tail log events from multiple hosts in realtime with devops friendly interface</p>

CC BY-SA 3.0
2017-10-23T04:27:58.217Z
2017-10-23T04:39:11.513Z
2017-10-23T04:39:11.513Z
Wissam Roujoulah
<h1>centralized <code>tmux</code></h1>

<p>While the other answers are smarter and wiser for the long term I think the quick hacky CLI solution is worth mentioning.  Run <code>tmux</code> on one server that can reach all of the others.  A good place for this would be a jump box or some other place that folks are commonly logged in anyway.  Within this "central" <code>tmux</code> ssh to each box in a different pane and tail whatever log files are necessary.  You can use <kbd>ctrl</kbd>-<kbd>b</kbd> <kbd>"</kbd> to get more panes in one tab within <code>tmux</code>.  Now all someone has to do to check things is attach to the "central" <code>tmux</code> session and they can see the whole cluster at a glance.</p>

<p>I spent a lot of time building the web UI solutions that you are working toward, but if you need it today hacking together something with <code>tmux</code> can save the day.</p>

CC BY-SA 3.0
2017-10-24T17:39:02.013Z
2017-10-24T17:39:02.013Z
chicks
<p>Any time you're running ad-hoc long-running commands, you should step back and rethink your process, because that should be automated, including error handling.</p>

<p>Rather than connecting in to the servers to see status, a better approach is to push that information out. You can do a wide variety of things if your want to write a bunch of custom code, but the simplest thing is probably to start sending the output through syslog to a centralized logging system (syslog itself, or ELK, or whatever). That way you can monitor everything from a central location.</p>

<p>Thinking forward, if this isn't a one-off task, monitoring should be automated. That is, you should never have to just watch logs to see if things are progressing as they're supposed to. Instead, you should assume they are (and continue with other work) <em>until your alerting fires off</em>. This is an investment of time into getting reliable and wide-coverage alerting, but as your systems grow in complexity, it will pay off as you don't have to monitor <strong>everything</strong> any time you change <strong>anything</strong>.</p>

CC BY-SA 3.0
2017-10-22T12:36:37.923Z
This is not a one off thing. I like your idea about investing time into automation of monitoring and centralizing the logging. Do you have any suggestions for tools that are free to use and that works well with ubuntu hosts running the programs?
guano
CC BY-SA 3.0
2017-10-23T19:26:26.143Z
@guano I think Wissam has [covered all the specific tools](https://devops.stackexchange.com/a/2391/960) I would mention, aside from using something like Sensu to power the alerting.
Xiong Chiamiov
CC BY-SA 3.0
2017-10-21T15:46:00.110Z
2017-10-21T15:46:00.110Z
Xiong Chiamiov
<p>We have three servers that are running python programs that are running data analysis tasks inside a <code>tmux</code> session. The method we are using at the moment is ssh'ing into each of them connecting the <code>tmux</code> session and watching the output on the command line. </p>

<p>This method is tedious, so what we are looking for is a solution that automates monitoring of program progress(output on CLI) for multiple servers at the same time. We would ideally like a web UI solution but a CLI would also be perfectly suitable.</p>

<p>Thank you for reading.</p>

CC BY-SA 3.0
2017-10-28T19:21:10.410Z
Use [tag:prometheus] and grafana :-)
Martin Schröder
<monitoring>
Monitor program progress on multiple servers
guano
# Monitor program progress on multiple servers

(Asked by: username_0 on 2017-10-21)

We have three servers that are running python programs that are running data analysis tasks inside a `tmux` session. The method we are using at the moment is ssh'ing into each of them connecting the `tmux` session and watching the output on the command line.

This method is tedious, so what we are looking for is a solution that automates monitoring of program progress(output on CLI) for multiple servers at the same time. We would ideally like a web UI solution but a CLI would also be perfectly suitable.

Thank you for reading.

**Tags:** monitoring

> **Comments:**
> **username_1**: Use [tag:prometheus] and grafana :-)  

---

## 4 answers

### Answer by username_5 on 2017-10-21. Score: 8 | Accepted answer

Any time you're running ad-hoc long-running commands, you should step back and rethink your process, because that should be automated, including error handling.

Rather than connecting in to the servers to see status, a better approach is to push that information out. You can do a wide variety of things if your want to write a bunch of custom code, but the simplest thing is probably to start sending the output through syslog to a centralized logging system (syslog itself, or ELK, or whatever). That way you can monitor everything from a central location.

Thinking forward, if this isn't a one-off task, monitoring should be automated. That is, you should never have to just watch logs to see if things are progressing as they're supposed to. Instead, you should assume they are (and continue with other work) *until your alerting fires off*. This is an investment of time into getting reliable and wide-coverage alerting, but as your systems grow in complexity, it will pay off as you don't have to monitor **everything** any time you change **anything**.

> **Comments:**
> **username_0**: This is not a one off thing. I like your idea about investing time into automation of monitoring and centralizing the logging. Do you have any suggestions for tools that are free to use and that works well with ubuntu hosts running the programs?  
> **username_5**: @username_0 I think Wissam has [covered all the specific tools](https://devops.stackexchange.com/a/2391/960) I would mention, aside from using something like Sensu to power the alerting.  

---

### Answer by username_2 on 2017-10-25. Score: 4

## Graylog

Since two people already advised you to rethink your current process (which I second since it will cause you sleepless nights at some point ;)), I will go another route and recommend a specific piece of software which - in my opinion - fits most of your needs: [Graylog](https://www.graylog.org/).

I implemented and used a couple of [ELK stacks](https://www.elastic.co/webinars/introduction-elk-stack) for both log aggregation and business intelligence and also run/maintain graylog for about two years now at my current employer. I recommend graylog since it has the following features built-in and is - in my opinion - a bit easier to setup and maintain:

* A web-interface
* Multiuser capabilities
* Alerting

As far as I understand your scenario, it looks as if you need to act or be alerted on certain events that appear in your stream of log messages. If we look at the [Graylog features](https://www.graylog.org/features):

> Trigger actions or get notified when something needs attention, such as failed login attempts, exceptions or performance degradation.
>
> Ideas: Send an email or Slack message to your team. Spawn a new machine to balance the processing load. Block IP ranges in your firewalls automatically when an attack is detected.

To give graylog a try I'd recommend the following two steps:

* Setup a dedicated host which is reachable by all you application hosts to run graylog (and its dependencies MongoDB and ElasticSearch)
* Send logs from your application to graylog (possibly as [GELF](http://docs.graylog.org/en/2.3/pages/gelf.html) messages)

Note: These two steps have the ability to fill pages and pages of best practices and should receive at least a couple of thoughts. Not to mention that graylog isn't a monitoring solution and graylog itself should be monitored by a proper monitoring tool (like e.g. Icinga, Prometheus, Nagios to name just a few).


---

### Answer by username_3 on 2017-10-23. Score: 3

I agree with @username_5 and I want to give more clarify option.
If you want every line in the CLI to be monitored, I would suggest to redirect all the output to specific file and the error to another file, then use [logstash](https://www.elastic.co/products/logstash) or [filebeat](https://www.elastic.co/products/beats/filebeat) to send both of this files to [Elasticsearch](https://www.elastic.co/products/elasticsearch), then you can configure [Logtril](https://github.com/sivasamyk/logtrail) with [Kibana](https://www.elastic.co/products/kibana) to give you view, analyze, search and tail log events from multiple hosts in realtime with devops friendly interface


---

### Answer by username_4 on 2017-10-24. Score: 1

# centralized `tmux`

While the other answers are smarter and wiser for the long term I think the quick hacky CLI solution is worth mentioning. Run `tmux` on one server that can reach all of the others. A good place for this would be a jump box or some other place that folks are commonly logged in anyway. Within this "central" `tmux` ssh to each box in a different pane and tail whatever log files are necessary. You can use `ctrl`-`b` `"` to get more panes in one tab within `tmux`. Now all someone has to do to check things is attach to the "central" `tmux` session and they can see the whole cluster at a glance.

I spent a lot of time building the web UI solutions that you are working toward, but if you need it today hacking together something with `tmux` can save the day.


---

