<p>There could be multiple ways to solve this.</p>
<ol>
<li><p>Create a python based cronjob that queries the DB every T secs (scraping interval) for the failed jobs in the last T secs (delta). Push the relevant data into a specific index like <code>failed-job</code> in ES. Then use that index to set up alerts in Grafana/opsgenie.(You can also use <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/how-watcher-works.html" rel="nofollow noreferrer">watcher</a> here) Don't need to read logs.</p>
</li>
<li><p>Use <a href="https://prometheus.io/" rel="nofollow noreferrer">prometheus</a>,(if already using), create a new <a href="https://prometheus.io/docs/instrumenting/writing_exporters/" rel="nofollow noreferrer">custom prometheus  exporter</a>.Exporter would be a Flask web server that provides an endpoint to run the query against the db and return results in a <a href="https://github.com/prometheus/client_python" rel="nofollow noreferrer">prometheus compatible format</a> to be stored. Configure job in prometheus to scrape this endpoint every T secs. Again, use Grafana for visualization.</p>
</li>
<li><p>Use a component called , <a href="https://prometheus.io/docs/instrumenting/pushing/" rel="nofollow noreferrer">prometheus-pushgateway</a>, if possible, either push the job status into this component using the python script as in option 1. or modify the Job itself to push the data directly. Again , can be connected to Prometheus/Grafana.</p>
</li>
</ol>
<p>If you are trying to solve only a single use case as described here, option 1 would be the better choice. For a larger more complex setups that covers a lot of other use cases, Prometheus might be worth looking into.</p>
<p>Alternatively, you can use the same design to push metrics into any datasource.(Like the one you are using for storing CPU/mem metrics, as specified.)</p>

CC BY-SA 4.0
2021-07-07T20:52:59.790Z
2021-07-07T20:52:59.790Z
rohatgisanat
<p>Not sure if it checks all your boxes but Rundeck is great for having an interface for your jobs . It alerts you if a job fails , has a quick summary for the jobs. <a href="https://docs.rundeck.com/docs/manual/01-introduction.html#who-makes-rundeck" rel="nofollow noreferrer">here</a> you can a look for details.</p>

CC BY-SA 4.0
2021-07-01T19:34:12.443Z
2021-07-01T19:34:12.443Z
gorboroth
<p>Maybe you can connect Grafana to the MySQL database and query for number of failures? Grafana also has a basic alerting capabilities, so you could send alerts to the interested parties.</p>
<p>At my company, we have a similar approach to jobs - the statuses are kept in the database. What we do to check for failures is we use Nagios and a custom query to check for problems and send alerts - but you might want to stick with Grafana and not install another tool.</p>

CC BY-SA 4.0
2021-07-02T08:43:31.340Z
2021-07-02T08:43:31.340Z
LL1138
<p>I would like to monitor for recent spike in failed application jobs at my company.</p>
<p>Job statuses are stored in a MySQL table.  We have on order of 50 different types of jobs, and sometimes, one of those jobs will start to fail, either because of a change in code or a change to a contract.  In Production we'd like to visualize the spike in failures and alert on it.</p>
<p>We have ELK, grafana, opsgenie, and python, bash scripting capabilities.
ELK doesn't seem to work because it parses logs, and our logs are not well standardized that we can detect a spike in failures.  Grafana seems to work better for metrics like CPU and memory.  We've done montioring in the past, by scripting, but surely there must be a better way.  Customers are hit with the problem long before we know about it.</p>
<p>Is there something in ELK, Grafana, or another open source tool that can help us here?</p>

<monitoring>
What is the best way to monitor job status?
MedicineMan
# What is the best way to monitor job status?

(Asked by: username_0 on 2021-07-01)

I would like to monitor for recent spike in failed application jobs at my company.

Job statuses are stored in a MySQL table. We have on order of 50 different types of jobs, and sometimes, one of those jobs will start to fail, either because of a change in code or a change to a contract. In Production we'd like to visualize the spike in failures and alert on it.

We have ELK, grafana, opsgenie, and python, bash scripting capabilities.
ELK doesn't seem to work because it parses logs, and our logs are not well standardized that we can detect a spike in failures. Grafana seems to work better for metrics like CPU and memory. We've done montioring in the past, by scripting, but surely there must be a better way. Customers are hit with the problem long before we know about it.

Is there something in ELK, Grafana, or another open source tool that can help us here?

**Tags:** monitoring


---

## 3 answers

### Answer by username_1 on 2021-07-07. Score: 2

There could be multiple ways to solve this.

1. Create a python based cronjob that queries the DB every T secs (scraping interval) for the failed jobs in the last T secs (delta). Push the relevant data into a specific index like `failed-job` in ES. Then use that index to set up alerts in Grafana/opsgenie.(You can also use [watcher](https://www.elastic.co/guide/en/elasticsearch/reference/current/how-watcher-works.html) here) Don't need to read logs.
2. Use [prometheus](https://prometheus.io/),(if already using), create a new [custom prometheus exporter](https://prometheus.io/docs/instrumenting/writing_exporters/).Exporter would be a Flask web server that provides an endpoint to run the query against the db and return results in a [prometheus compatible format](https://github.com/prometheus/client_python) to be stored. Configure job in prometheus to scrape this endpoint every T secs. Again, use Grafana for visualization.
3. Use a component called , [prometheus-pushgateway](https://prometheus.io/docs/instrumenting/pushing/), if possible, either push the job status into this component using the python script as in option 1. or modify the Job itself to push the data directly. Again , can be connected to Prometheus/Grafana.

If you are trying to solve only a single use case as described here, option 1 would be the better choice. For a larger more complex setups that covers a lot of other use cases, Prometheus might be worth looking into.

Alternatively, you can use the same design to push metrics into any datasource.(Like the one you are using for storing CPU/mem metrics, as specified.)


---

### Answer by username_2 on 2021-07-01. Score: 0

Not sure if it checks all your boxes but Rundeck is great for having an interface for your jobs . It alerts you if a job fails , has a quick summary for the jobs. [here](https://docs.rundeck.com/docs/manual/01-introduction.html#who-makes-rundeck) you can a look for details.


---

### Answer by username_3 on 2021-07-02. Score: 0

Maybe you can connect Grafana to the MySQL database and query for number of failures? Grafana also has a basic alerting capabilities, so you could send alerts to the interested parties.

At my company, we have a similar approach to jobs - the statuses are kept in the database. What we do to check for failures is we use Nagios and a custom query to check for problems and send alerts - but you might want to stick with Grafana and not install another tool.


---

