<p>This is more of an opinion based question but let me try to answer it.</p>
<blockquote>
<p>installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing.Am I right or just got a wrong idea from my research?</p>
</blockquote>
<blockquote>
<p>Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor?</p>
</blockquote>
<p>This is a correct practice to have. Monitoring systems are usually setup to look into the object being monitored. (<em>outside looking in</em>). But with self healing systems like k8s, installing monitoring within the system can be done with minimum issues. There are a few cases you do need to look into for proper stability.</p>
<ul>
<li>Pods get deleted :- Run Prometheus in a HA mode (2 replicas minimum)
so as not to lose any data.</li>
<li>Pods are replaced due to resource crunch:- Add <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass" rel="nofollow noreferrer">priority classes</a>
to prevent pod form being evicted.</li>
<li>Use Persistent volumes:- Look into <a href="https://stackoverflow.com/questions/51946393/kubernetes-pod-warning-1-nodes-had-volume-node-affinity-conflict">volume node affinity
conflict</a>. Also configure the data retention period to make
optimised use of storage.</li>
<li>Cluster goes down. See below.</li>
</ul>
<blockquote>
<p>Also, if I set up multiple Kubernetes clusters, isn't it more cumbersome to have one Prometheus installation per cluster?</p>
</blockquote>
<p>Not at all, using a combination of IaC(Terraform/Chef),Config management(Ansible),CI/CD (Spinnaker),GitOps and Helm charts. Configuring/maintaining  Prometheus instances across multiple instances will not be an issue.</p>
<blockquote>
<p>Is it possible to set up a single Prometheus installation and centralise all k8s clusters metrics on it?</p>
</blockquote>
<p>Prometheus although highly optimised, will not be able to handle all the metrics thrown at it from multiple clusters.Also Prometheus does not scale horizontally on its own, therefore it would need huge resources. Centralising all metrics to one place will cause other issues as well.</p>
<ul>
<li>Network Connectivity/Latency:- Running a centralised Prometheus
instance receiving data from over the public internet may cause data
loss/false alerts due to network drops and latency.</li>
<li>Cost:- Running tens of exporter all sending data over the internet
will cause high internet costs for cloud deployments.</li>
<li>Cluster managing:- You may face issues with exporter/job
configuration of such a large number of exporters.</li>
</ul>
<p>Its a good choice to run Prometheus per cluster. wherein you can have all the data locally till the time is required.</p>
<p>Grafana does not consume much resources so its safe to run inside the cluster. However it faces a different set of issues:-</p>
<ul>
<li>User/Permission Management :- Each local instance would need to be
maintained separately which is a huge task.</li>
<li>Charts Management:- Same as above.</li>
<li>L1/DevOps will have to keep n number of tabs open to monitor your
infra.</li>
</ul>
<p>Other than this, setting up alerts on all the clusters separately would also be a never-ending task.</p>
<p>My advice would be to setup Prometheus clusters locally (inside the cluster or right next to it) and connect all of them via <a href="https://thanos.io/" rel="nofollow noreferrer">Thanos</a> or <a href="https://cortexmetrics.io/" rel="nofollow noreferrer">Cortex</a>
and set up a HA mode Grafana centrally on top of Thanos along with alerts and external storage.This will alert if the local Prometheus instance goes down and will help with the <em>outside looking in</em> factor</p>
<blockquote>
<p>scenarios in which a longer history period might help. Any comments on this matter?</p>
</blockquote>
<p>Depends upon the metrics being stored, A quarterly SLI/SLO and incident reports seem like a good use case to store data for longer than 15 days.</p>

CC BY-SA 4.0
2021-05-25T20:36:28.570Z
2021-05-25T20:36:28.570Z
rohatgisanat
<p>I'm considering using Prometheus to monitor and analyze different metrics from my AWS EKS Kubernetes cluster running in Fargate, as it seems to be the standard tool during these times. However, something that really impressed me was that installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing. Am I right or just got a wrong idea from my research? Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor? Also, if I set up multiple Kubernetes clusters, isn't it more cumbersome to have one Prometheus installation per cluster? Is it possible to set up a single Prometheus installation and centralize all k8s clusters metrics on it?</p>
<p>Same question for Grafana's installation... Should it be installed inside or outside the cluster?</p>
<p>On a different topic, Prometheus history retention period is set to 15 days by default (<code>storage.tsdb.retention.time</code>). Even though cluster metrics insight should be useful for a short period of time so you can troubleshoot issues, I wonder if there may be scenarios in which a longer history period might help. Any comments on this matter?</p>
<p>Thanks!</p>

<kubernetes><monitoring><prometheus><grafana><eks>
Where to install Prometheus for Kubernetes monitoring
Nicolás García
# Where to install Prometheus for Kubernetes monitoring

(Asked by: username_0 on 2021-05-10)

I'm considering using Prometheus to monitor and analyze different metrics from my AWS EKS Kubernetes cluster running in Fargate, as it seems to be the standard tool during these times. However, something that really impressed me was that installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing. Am I right or just got a wrong idea from my research? Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor? Also, if I set up multiple Kubernetes clusters, isn't it more cumbersome to have one Prometheus installation per cluster? Is it possible to set up a single Prometheus installation and centralize all k8s clusters metrics on it?

Same question for Grafana's installation... Should it be installed inside or outside the cluster?

On a different topic, Prometheus history retention period is set to 15 days by default (`storage.tsdb.retention.time`). Even though cluster metrics insight should be useful for a short period of time so you can troubleshoot issues, I wonder if there may be scenarios in which a longer history period might help. Any comments on this matter?

Thanks!

**Tags:** kubernetes, monitoring, prometheus, grafana, eks


---

## 1 answers

### Answer by username_1 on 2021-05-25. Score: 2

This is more of an opinion based question but let me try to answer it.

> installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing.Am I right or just got a wrong idea from my research?

> Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor?

This is a correct practice to have. Monitoring systems are usually setup to look into the object being monitored. (*outside looking in*). But with self healing systems like k8s, installing monitoring within the system can be done with minimum issues. There are a few cases you do need to look into for proper stability.

* Pods get deleted :- Run Prometheus in a HA mode (2 replicas minimum)
  so as not to lose any data.
* Pods are replaced due to resource crunch:- Add [priority classes](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass)
  to prevent pod form being evicted.
* Use Persistent volumes:- Look into [volume node affinity
  conflict](https://stackoverflow.com/questions/51946393/kubernetes-pod-warning-1-nodes-had-volume-node-affinity-conflict). Also configure the data retention period to make
  optimised use of storage.
* Cluster goes down. See below.

> Also, if I set up multiple Kubernetes clusters, isn't it more cumbersome to have one Prometheus installation per cluster?

Not at all, using a combination of IaC(Terraform/Chef),Config management(Ansible),CI/CD (Spinnaker),GitOps and Helm charts. Configuring/maintaining Prometheus instances across multiple instances will not be an issue.

> Is it possible to set up a single Prometheus installation and centralise all k8s clusters metrics on it?

Prometheus although highly optimised, will not be able to handle all the metrics thrown at it from multiple clusters.Also Prometheus does not scale horizontally on its own, therefore it would need huge resources. Centralising all metrics to one place will cause other issues as well.

* Network Connectivity/Latency:- Running a centralised Prometheus
  instance receiving data from over the public internet may cause data
  loss/false alerts due to network drops and latency.
* Cost:- Running tens of exporter all sending data over the internet
  will cause high internet costs for cloud deployments.
* Cluster managing:- You may face issues with exporter/job
  configuration of such a large number of exporters.

Its a good choice to run Prometheus per cluster. wherein you can have all the data locally till the time is required.

Grafana does not consume much resources so its safe to run inside the cluster. However it faces a different set of issues:-

* User/Permission Management :- Each local instance would need to be
  maintained separately which is a huge task.
* Charts Management:- Same as above.
* L1/DevOps will have to keep n number of tabs open to monitor your
  infra.

Other than this, setting up alerts on all the clusters separately would also be a never-ending task.

My advice would be to setup Prometheus clusters locally (inside the cluster or right next to it) and connect all of them via [Thanos](https://thanos.io/) or [Cortex](https://cortexmetrics.io/)
and set up a HA mode Grafana centrally on top of Thanos along with alerts and external storage.This will alert if the local Prometheus instance goes down and will help with the *outside looking in* factor

> scenarios in which a longer history period might help. Any comments on this matter?

Depends upon the metrics being stored, A quarterly SLI/SLO and incident reports seem like a good use case to store data for longer than 15 days.


---

