<p>I'm not sure it's a best practice but what I would do is using <a href="https://www.elastic.co/products/beats/filebeat" rel="noreferrer">filebeat</a> to read the log in near real time and push it to ElasticSearch.<br>
AWS provides an <a href="https://aws.amazon.com/elasticsearch-service/?nc2=h_l3_al" rel="noreferrer">ElasticSearch Service</a> which can save you from this part.</p>

<p>The drawback of using filebeat directly to elasticsearch is that the time of events will be the time the prospector read them and if your logs have a timestamp they can be "ingested" in different order than produced (from multiple workers).</p>

<p>You can add <a href="https://www.elastic.co/products/logstash" rel="noreferrer">Logstash</a> inbetween, sending from filebeat to logstash and then to elasticsearch. Logstash is able to interpret your logs and split them into fields to search again, in the same manner you can parse your date in log so events will be stored with a correct timestamp.</p>

<p>You may use logstash directly to read the logs, but it's quite heavy on resources so I would keep it separated to no impact the workers.</p>

<p>If you buy a license from x-pack you can add alerts on messages directly from elasticsearch.</p>

<p>If you want to save you from the burden of maintaining logstash there's services like <a href="https://logmatic.io/" rel="noreferrer">logmatic.io</a> which provides you with the full stack, you just have to push your logs and a bunch of parsing are handled from stock.</p>

CC BY-SA 3.0
2017-04-03T08:07:38.087Z
2017-04-03T08:07:38.087Z
Tensibai
<p>I use <code>nohup</code> for starting Ansible playbooks inside the Master which is spawned via boto3 from AWS Lambda.</p>

<p>This playbook creates multiple workers, make them do some tasks, and terminates them.</p>

<p>So, I want to design/build a reliable monitoring system for the same. For now, I am writing the logs of <code>nohup</code> to a <code>tasks.log</code> file inside master, and pushing it to a store(for now, S3) once the Ansible plays are completed.</p>

<p><strong>But, I wanted to know whether there is a much more elegant (and/or) better method for doing the same?  And also, whether there are good tools which would help me in the process?</strong></p>

CC BY-SA 3.0
2017-03-31T09:59:20.700Z
would something around ELK/filebeat to push the logs be an interesting option ?
Tensibai
CC BY-SA 3.0
2017-03-31T10:05:49.907Z
@Tensibai Yeah, we're exploring the ELK stack too :) . But, for now, S3 is sufficing.
Dawny33
CC BY-SA 3.0
2017-03-31T10:12:06.510Z
I'm very unsure to understand what you're after in fact :p
Tensibai
CC BY-SA 3.0
2017-04-02T04:15:49.183Z
What do you want to know about them?
Xiong Chiamiov
CC BY-SA 3.0
2017-06-21T18:52:05.257Z
It seems silly to use `nohup` when redirection to a log file and running in the background are built-in to the shell these days.
chicks
<ansible><aws-lambda><monitoring><practices>
Are there any best practices and/or tools for 'nohup' monitoring?
Dawny33
# Are there any best practices and/or tools for 'nohup' monitoring?

(Asked by: username_0 on 2017-03-31)

I use `nohup` for starting Ansible playbooks inside the Master which is spawned via boto3 from AWS Lambda.

This playbook creates multiple workers, make them do some tasks, and terminates them.

So, I want to design/build a reliable monitoring system for the same. For now, I am writing the logs of `nohup` to a `tasks.log` file inside master, and pushing it to a store(for now, S3) once the Ansible plays are completed.

**But, I wanted to know whether there is a much more elegant (and/or) better method for doing the same? And also, whether there are good tools which would help me in the process?**

**Tags:** ansible, aws-lambda, monitoring, practices

> **Comments:**
> **username_1**: would something around ELK/filebeat to push the logs be an interesting option ?  
> **username_0**: @username_1 Yeah, we're exploring the ELK stack too :) . But, for now, S3 is sufficing.  
> **username_1**: I'm very unsure to understand what you're after in fact :p  
> **username_2**: What do you want to know about them?  
> **username_3**: It seems silly to use `nohup` when redirection to a log file and running in the background are built-in to the shell these days.  

---

## 1 answers

### Answer by username_1 on 2017-04-03. Score: 8 | Accepted answer

I'm not sure it's a best practice but what I would do is using [filebeat](https://www.elastic.co/products/beats/filebeat) to read the log in near real time and push it to ElasticSearch.  
AWS provides an [ElasticSearch Service](https://aws.amazon.com/elasticsearch-service/?nc2=h_l3_al) which can save you from this part.

The drawback of using filebeat directly to elasticsearch is that the time of events will be the time the prospector read them and if your logs have a timestamp they can be "ingested" in different order than produced (from multiple workers).

You can add [Logstash](https://www.elastic.co/products/logstash) inbetween, sending from filebeat to logstash and then to elasticsearch. Logstash is able to interpret your logs and split them into fields to search again, in the same manner you can parse your date in log so events will be stored with a correct timestamp.

You may use logstash directly to read the logs, but it's quite heavy on resources so I would keep it separated to no impact the workers.

If you buy a license from x-pack you can add alerts on messages directly from elasticsearch.

If you want to save you from the burden of maintaining logstash there's services like [logmatic.io](https://logmatic.io/) which provides you with the full stack, you just have to push your logs and a bunch of parsing are handled from stock.


---

