<p>I found out what was happening in my graphs. All the Pods mentioned above have one thing in common: they use the host's network namespace, so their network metrics are <em>all identical</em>, and equal to the global <em>host's metric</em> (just with a slightly different precision).</p>
<pre><code>$ kubectl - monitoring get pod -o jsonpath='{.spec.hostNetwork}' \
    prometheus-stack-prometheus-node-exporter-jnhw7

true
</code></pre>
<pre><code>$ kubectl -n kube-system get pod -o jsonpath='{.items[*].spec.hostNetwork}' \
    kube-proxy-gke-triggermesh-product-control-plane-7fc0ad24-z586 \
    gke-metrics-agent-5cv4m \
    prometheus-to-sd-tk8jv \
    fluentbit-gke-xh879

true true true true 
</code></pre>
<p>One way to see it is to compare the host's metric to <em>one</em> of the above Pods:</p>
<p><a href="https://i.sstatic.net/RHQiT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RHQiT.png" alt="Comparison of eth0 and a Pod with hostNetwork" /></a></p>

CC BY-SA 4.0
2022-11-11T13:50:21.313Z
Thanks for the hint. We researched a similar problem today. Our node-exporter pod bandwith massively increased (more than 400mb/s) - which we could not explain. In our case a redis pod had very high network transmit, which was routed trough calico/kube-proxy in the hostNetwork.

Although the host machine had only 150mb/s network transmit, which was caused by calico which only routed external traffic to other nodes. Lots of traffic stayed inside the virtual network of the host itself.
Martin Lantzsch
CC BY-SA 4.0
2020-11-20T19:43:10.713Z
2020-11-20T19:43:10.713Z
Antoine Cotten
<h3>Context</h3>
<p>I am trying to chart the network bandwidth usage of a node in 2 different manners:</p>
<ol>
<li>By looking at global metrics for that node</li>
<li>By summing up the corresponding metric for each Pod</li>
</ol>
<p>To achieve this, I am issuing the following Prometheus queries (<em>example for the receive bandwidth</em>):</p>
<ul>
<li><p>For the entire node (<em>metric from node-exporter</em>)</p>
<pre><code>sum(irate(node_network_receive_bytes_total{instance=&quot;10.142.0.54:9100&quot;}[$__rate_interval])) by (device)
</code></pre>
</li>
<li><p>Per Pod (<em>metric from kubelet</em>)</p>
<pre><code>sum(irate(container_network_receive_bytes_total{node=&quot;$node&quot;,container!=&quot;&quot;}[$__rate_interval])) by (pod,interface)
</code></pre>
</li>
</ul>
<p>The results are displayed in the following Grafana dashboard, after generating some load on a HTTP service called <code>thrpt-receiver</code>:</p>
<p><a href="https://i.sstatic.net/sZtI5.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/sZtI5.png" alt="Receive bandwidth" /></a></p>
<p>Here's what I see if I look at the raw metrics, without <code>sum()</code> and <code>irate()</code> applied:</p>
<p><a href="https://i.sstatic.net/ASC5X.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ASC5X.png" alt="Received bytes" /></a></p>
<h3>Problem</h3>
<p>As you can see, results are vastly different, to the point I'm almost certain I am doing something wrong, but what?</p>
<p>What makes me especially suspicious about the Pod metrics is the supposedly increasing received bandwidth of kube-proxy (which AFAIK is not supposed to be receiving any traffic in iptables mode), and agents such as the Prometheus node-exporter, etc.</p>

<prometheus>
Node-centric network metrics are very different from the sum of metrics of the Pods running on that node
Antoine Cotten
# Node-centric network metrics are very different from the sum of metrics of the Pods running on that node

(Asked by: username_0 on 2020-11-20)

Context
I am trying to chart the network bandwidth usage of a node in 2 different manners:

By looking at global metrics for that node
By summing up the corresponding metric for each Pod

To achieve this, I am issuing the following Prometheus queries (example for the receive bandwidth):

For the entire node (metric from node-exporter)
sum(irate(node_network_receive_bytes_total{instance="10.142.0.54:9100"}[$__rate_interval])) by (device)


Per Pod (metric from kubelet)
sum(irate(container_network_receive_bytes_total{node="$node",container!=""}[$__rate_interval])) by (pod,interface)



The results are displayed in the following Grafana dashboard, after generating some load on a HTTP service called thrpt-receiver:

Here's what I see if I look at the raw metrics, without sum() and irate() applied:

Problem
As you can see, results are vastly different, to the point I'm almost certain I am doing something wrong, but what?
What makes me especially suspicious about the Pod metrics is the supposedly increasing received bandwidth of kube-proxy (which AFAIK is not supposed to be receiving any traffic in iptables mode), and agents such as the Prometheus node-exporter, etc.

**Tags:** prometheus


---

## 1 answers

### Answer by username_0 on 2020-11-20. Score: 1 | Accepted answer

I found out what was happening in my graphs. All the Pods mentioned above have one thing in common: they use the host's network namespace, so their network metrics are all identical, and equal to the global host's metric (just with a slightly different precision).
$ kubectl - monitoring get pod -o jsonpath='{.spec.hostNetwork}' \
    prometheus-stack-prometheus-node-exporter-jnhw7

true

$ kubectl -n kube-system get pod -o jsonpath='{.items[*].spec.hostNetwork}' \
    kube-proxy-gke-triggermesh-product-control-plane-7fc0ad24-z586 \
    gke-metrics-agent-5cv4m \
    prometheus-to-sd-tk8jv \
    fluentbit-gke-xh879

true true true true 

One way to see it is to compare the host's metric to one of the above Pods:

> **Comments:**
> **username_1**: Thanks for the hint. We researched a similar problem today. Our node-exporter pod bandwith massively increased (more than 400mb/s) - which we could not explain. In our case a redis pod had very high network transmit, which was routed trough calico/kube-proxy in the hostNetwork.  Although the host machine had only 150mb/s network transmit, which was caused by calico which only routed external traffic to other nodes. Lots of traffic stayed inside the virtual network of the host itself.  

---

