<p>This command <code>kubectl top pods --all-namespaces</code> is only displaying resource (CPU/memory) usage of pods, it's not related to other processes outside your cluster in your OS, so the total pods consumption memory <code>9146Mi</code> is fine.</p>
<p>But for your node, there is definitely something taking extra memory,  I think you might need to do some below checks:</p>
<ul>
<li><p><code>htop</code> will sort <em><strong>processes</strong></em> based on memory usage, and it might not give much info, because in Linux everything is either <em><strong>file</strong></em> or <em><strong>process</strong></em>. You might need to check also the <em><strong>files</strong></em> opened using <code>lsof</code> as they are also consuming memory.</p>
</li>
<li><p>Do you use a database proxy (e.g, Heimdall) in front of your node? If so, it might reserve some memory because of the open connections.</p>
</li>
<li><p>check how much memory assigned to K3s in the first place, does is it
reserve a fixed amount of memory?</p>
</li>
<li><p>It's worth to try to stop Gitlab process, and check the node's memory again.</p>
</li>
</ul>

CC BY-SA 4.0
2024-04-02T22:26:31.977Z
2024-04-02T22:27:30.480Z
2024-04-02T22:27:30.480Z
Faisal
<p>I'm facing a discrepancy in memory usage reporting in my Kubernetes cluster, and I'm hoping to gain some insights into what might be causing it.</p>
<p>When I run <code>kubectl top pods --all-namespaces</code>, I get the following output listing memory usage for various pods in the cluster:</p>
<pre><code>NAMESPACE           NAME                                                CPU(cores)   MEMORY(bytes)
argocd              argocd-application-controller-0                     40m          185Mi
argocd              argocd-applicationset-controller-584f68b9d7-l954j   1m           51Mi
argocd              argocd-dex-server-8577d9498b-l9lnv                  1m           17Mi
argocd              argocd-notifications-controller-564dcb4995-j582k    1m           25Mi
argocd              argocd-redis-66d9777b78-29mq6                       3m           4Mi
argocd              argocd-repo-server-58c94b5cbf-477v8                 1m           71Mi
argocd              argocd-server-b8bd4f4b5-kz4qw                       1m           41Mi
cert-manager        cert-manager-5b54fc556f-jjsqb                       1m           22Mi
cert-manager        cert-manager-cainjector-7d8b6cf7b9-885vx            1m           26Mi
cert-manager        cert-manager-webhook-7d4744b5ff-cmk98               1m           9Mi
gitlab              gitlab-deployment-66dcd84ffb-b58cs                  133m         6505Mi
kube-system         coredns-6799fbcd5-q7pns                             11m          17Mi
kube-system         local-path-provisioner-84db5d44d9-2466v             1m           7Mi
kube-system         metrics-server-67c658944b-44f5n                     12m          19Mi
kube-system         svclb-traefik-c1466e35-j2psx                        0m           2Mi
kube-system         traefik-5ff9bf948b-bh2bp                            1m           34Mi
linkerd             linkerd-destination-7b5d9869c8-69jwx                8m           63Mi
linkerd             linkerd-identity-7cfd498f58-jpdtr                   3m           24Mi
linkerd             linkerd-proxy-injector-f6c775c9c-wj9cj              3m           27Mi
linkerd-viz         grafana-5646d97bff-n7bj4                            14m          93Mi
linkerd-viz         metrics-api-5f5789c8cf-5mpzp                        2m           29Mi
linkerd-viz         prometheus-6c96bcf74d-85cs9                         25m          154Mi
linkerd-viz         tap-d94cf7688-np8sk                                 5m           29Mi
linkerd-viz         tap-injector-7c5b57889c-h8ft5                       2m           23Mi
linkerd-viz         web-69bbf5b8db-tz9t2                                5m           26Mi
microservice-demo   auth-service-deployment-69dc69b858-7zfl6            7m           543Mi
microservice-demo   backend-api-deployment-786b954f-7kbg4               18m          99Mi
microservice-demo   mongodb-deployment-7d94b89cf5-xs5gs                 12m          288Mi
microservice-demo   payout-service-deployment-7fdf698f68-78zzx          7m           79Mi
microservice-demo   wallet-service-deployment-687fc74bdb-hlh4w          11m          86Mi
microservice-demo   web-app-deployment-5dc5dcbf7c-lz9q8                 1m           7Mi
microservice-demo   wheel-service-deployment-5bfb59d776-nr4pf           4m           67Mi
</code></pre>
<p>Summing up the memory usage of these pods, the total comes to approximately 9146Mi.</p>
<p>However, when I check the memory usage of the node using kubectl top, it reports a much higher memory usage:</p>
<pre><code>NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
myserver  785m         9%     16508Mi         51%
</code></pre>
<p>As you can see, the memory usage reported for the node (16508Mi) is significantly higher than the sum of memory usage of the listed pods (9146Mi).</p>
<p>I have verified that the output of kubectl top pods --all-namespaces includes all pods running on the node, so I'm puzzled as to what might be consuming the additional memory reported for the node.</p>
<p>Using <code>htop</code> I also don't get any additional information, except the running pods, because <code>htop</code> tells me that GitLab is the one that is using the most:
<a href="https://i.sstatic.net/Ijx9d.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ijx9d.png" alt="enter image description here" /></a></p>
<p>Could anyone provide insights into why there might be such a discrepancy? What other factors or components could be contributing to the memory usage reported for the node and how I can decrease it. It feels like the whole k3s thing is using way too much memory in total for the clusters that I run. I see people running GitLab on raspberry pi's and stuff, so why is 16GB of RAM not enough?</p>

<kubernetes><k3s>
Discrepancy between Kubernetes node memory usage reported by kubectl top and summed pod memory usage
A. Vreeswijk
# Discrepancy between Kubernetes node memory usage reported by kubectl top and summed pod memory usage

(Asked by: username_0 on 2024-03-25)

I'm facing a discrepancy in memory usage reporting in my Kubernetes cluster, and I'm hoping to gain some insights into what might be causing it.
When I run kubectl top pods --all-namespaces, I get the following output listing memory usage for various pods in the cluster:
NAMESPACE           NAME                                                CPU(cores)   MEMORY(bytes)
argocd              argocd-application-controller-0                     40m          185Mi
argocd              argocd-applicationset-controller-584f68b9d7-l954j   1m           51Mi
argocd              argocd-dex-server-8577d9498b-l9lnv                  1m           17Mi
argocd              argocd-notifications-controller-564dcb4995-j582k    1m           25Mi
argocd              argocd-redis-66d9777b78-29mq6                       3m           4Mi
argocd              argocd-repo-server-58c94b5cbf-477v8                 1m           71Mi
argocd              argocd-server-b8bd4f4b5-kz4qw                       1m           41Mi
cert-manager        cert-manager-5b54fc556f-jjsqb                       1m           22Mi
cert-manager        cert-manager-cainjector-7d8b6cf7b9-885vx            1m           26Mi
cert-manager        cert-manager-webhook-7d4744b5ff-cmk98               1m           9Mi
gitlab              gitlab-deployment-66dcd84ffb-b58cs                  133m         6505Mi
kube-system         coredns-6799fbcd5-q7pns                             11m          17Mi
kube-system         local-path-provisioner-84db5d44d9-2466v             1m           7Mi
kube-system         metrics-server-67c658944b-44f5n                     12m          19Mi
kube-system         svclb-traefik-c1466e35-j2psx                        0m           2Mi
kube-system         traefik-5ff9bf948b-bh2bp                            1m           34Mi
linkerd             linkerd-destination-7b5d9869c8-69jwx                8m           63Mi
linkerd             linkerd-identity-7cfd498f58-jpdtr                   3m           24Mi
linkerd             linkerd-proxy-injector-f6c775c9c-wj9cj              3m           27Mi
linkerd-viz         grafana-5646d97bff-n7bj4                            14m          93Mi
linkerd-viz         metrics-api-5f5789c8cf-5mpzp                        2m           29Mi
linkerd-viz         prometheus-6c96bcf74d-85cs9                         25m          154Mi
linkerd-viz         tap-d94cf7688-np8sk                                 5m           29Mi
linkerd-viz         tap-injector-7c5b57889c-h8ft5                       2m           23Mi
linkerd-viz         web-69bbf5b8db-tz9t2                                5m           26Mi
microservice-demo   auth-service-deployment-69dc69b858-7zfl6            7m           543Mi
microservice-demo   backend-api-deployment-786b954f-7kbg4               18m          99Mi
microservice-demo   mongodb-deployment-7d94b89cf5-xs5gs                 12m          288Mi
microservice-demo   payout-service-deployment-7fdf698f68-78zzx          7m           79Mi
microservice-demo   wallet-service-deployment-687fc74bdb-hlh4w          11m          86Mi
microservice-demo   web-app-deployment-5dc5dcbf7c-lz9q8                 1m           7Mi
microservice-demo   wheel-service-deployment-5bfb59d776-nr4pf           4m           67Mi

Summing up the memory usage of these pods, the total comes to approximately 9146Mi.
However, when I check the memory usage of the node using kubectl top, it reports a much higher memory usage:
NAME      CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
myserver  785m         9%     16508Mi         51%

As you can see, the memory usage reported for the node (16508Mi) is significantly higher than the sum of memory usage of the listed pods (9146Mi).
I have verified that the output of kubectl top pods --all-namespaces includes all pods running on the node, so I'm puzzled as to what might be consuming the additional memory reported for the node.
Using htop I also don't get any additional information, except the running pods, because htop tells me that GitLab is the one that is using the most:

Could anyone provide insights into why there might be such a discrepancy? What other factors or components could be contributing to the memory usage reported for the node and how I can decrease it. It feels like the whole k3s thing is using way too much memory in total for the clusters that I run. I see people running GitLab on raspberry pi's and stuff, so why is 16GB of RAM not enough?

**Tags:** kubernetes, k3s


---

## 1 answers

### Answer by username_1 on 2024-04-02. Score: 0

This command `kubectl top pods --all-namespaces` is only displaying resource (CPU/memory) usage of pods, it's not related to other processes outside your cluster in your OS, so the total pods consumption memory `9146Mi` is fine.

But for your node, there is definitely something taking extra memory, I think you might need to do some below checks:

* `htop` will sort ***processes*** based on memory usage, and it might not give much info, because in Linux everything is either ***file*** or ***process***. You might need to check also the ***files*** opened using `lsof` as they are also consuming memory.
* Do you use a database proxy (e.g, Heimdall) in front of your node? If so, it might reserve some memory because of the open connections.
* check how much memory assigned to K3s in the first place, does is it
  reserve a fixed amount of memory?
* It's worth to try to stop Gitlab process, and check the node's memory again.


---

