<p>Based on what I see so far, I have found the root cause, renamed kubelet service from old chart to new that can get targeted by serviceMonitors. So for me the best solution would be grafana kube-state-metrics + comparing what I see in the jvm dashboard</p>

CC BY-SA 4.0
2020-10-15T15:17:52.810Z
2020-10-15T15:17:52.810Z
anVzdGFub3RoZXJodW1hbg
<p>micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -&gt; K8s Mixin from kube-promethteus-stack Grafana dashboad
It's really blurry and frustrating to me to understand why there is such big difference between values from the 3 in the title and how should one utilize K8s Mixin to set proper request/limits and if that is expected at al.
I was hoping I can just see same data that I see when I type kubectl top podname --containers to what I see when I open K8s -&gt; ComputeResources -&gt; Pods dashboard in Grafana. But not only the values differ by more than a double, but also reported values from actuator differ from both.
When exposing spring data with micrometer the sum of jvm_memory_used_bytes is corresponding more to what I get from metrics-server (0.37.0) rather then what I see on Grafana from the mixin dashboards, but it is still far off.
I am using K8s: 1.14.3 on Ubuntu 18.04 LTS managed by kubespray.
kube-prometheus-stack 9.4.4 installed with helm 2.14.3.
Spring boot 2.0 with Micrometer. I saw the explanation on metrics-server git that this is the value that kubelet use for OOMKill, but again this is not helpful at all as what should I do with the dashboard? What is the the way to handle this?</p>
<p><a href="https://i.sstatic.net/aVzeS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aVzeS.png" alt="enter image description here" /></a></p>
<p><a href="https://i.sstatic.net/elHG4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/elHG4.png" alt="enter image description here" /></a></p>
<p><a href="https://i.sstatic.net/m1DZm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m1DZm.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2020-10-15T14:42:26.897Z
I found out the double values are result of something in the provided querry in Grafana: When I run the following instant vector container_memory_working_set_bytes{cluster="", namespace="qa2", pod="service-x-684f5565b7-wbgm6", container!="POD", container!="", image!=""} why would I get 4 results instead of 2 (service and istio-proxy), what is the logical explanation behind this, I get 2 time the service and 2 time the proxy with same values, but different id in the metrics endpoint is that a bug in kube-prometheus-stack helm chart or perhaps something on operator level?
anVzdGFub3RoZXJodW1hbg
<kubernetes><prometheus><java><grafana>
micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits
anVzdGFub3RoZXJodW1hbg
# micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits

(Asked by: username_0 on 2020-10-15)

micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -> K8s Mixin from kube-promethteus-stack Grafana dashboad
It's really blurry and frustrating to me to understand why there is such big difference between values from the 3 in the title and how should one utilize K8s Mixin to set proper request/limits and if that is expected at al.
I was hoping I can just see same data that I see when I type kubectl top podname --containers to what I see when I open K8s -> ComputeResources -> Pods dashboard in Grafana. But not only the values differ by more than a double, but also reported values from actuator differ from both.
When exposing spring data with micrometer the sum of jvm\_memory\_used\_bytes is corresponding more to what I get from metrics-server (0.37.0) rather then what I see on Grafana from the mixin dashboards, but it is still far off.
I am using K8s: 1.14.3 on Ubuntu 18.04 LTS managed by kubespray.
kube-prometheus-stack 9.4.4 installed with helm 2.14.3.
Spring boot 2.0 with Micrometer. I saw the explanation on metrics-server git that this is the value that kubelet use for OOMKill, but again this is not helpful at all as what should I do with the dashboard? What is the the way to handle this?

[![enter image description here](https://i.sstatic.net/aVzeS.png)](https://i.sstatic.net/aVzeS.png)

[![enter image description here](https://i.sstatic.net/elHG4.png)](https://i.sstatic.net/elHG4.png)

[![enter image description here](https://i.sstatic.net/m1DZm.png)](https://i.sstatic.net/m1DZm.png)

**Tags:** kubernetes, prometheus, java, grafana

> **Comments:**
> **username_0**: I found out the double values are result of something in the provided querry in Grafana: When I run the following instant vector container_memory_working_set_bytes{cluster="", namespace="qa2", pod="service-x-684f5565b7-wbgm6", container!="POD", container!="", image!=""} why would I get 4 results instead of 2 (service and istio-proxy), what is the logical explanation behind this, I get 2 time the service and 2 time the proxy with same values, but different id in the metrics endpoint is that a bug in kube-prometheus-stack helm chart or perhaps something on operator level?  

---

## 1 answers

### Answer by username_0 on 2020-10-15. Score: 0

Based on what I see so far, I have found the root cause, renamed kubelet service from old chart to new that can get targeted by serviceMonitors. So for me the best solution would be grafana kube-state-metrics + comparing what I see in the jvm dashboard


---

