<p>To piggy back off of @eduardo-baitello answer, A third option is to increase the <code>CPUThrottlingPercent</code> config <a href="https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/f7ca48cca5d9cadc9a2203b8c0b3bb3eb85f3294/config.libsonnet#L44" rel="nofollow noreferrer">here</a></p>

CC BY-SA 4.0
2021-02-08T23:28:40.710Z
Mind posting this as a comment?
codeforester
CC BY-SA 4.0
2021-02-08T19:12:58.913Z
2021-02-08T19:12:58.913Z
Saleh Rastani
<p>The <code>CPUThrottlingHigh</code> is an alert created by the <a href="https://github.com/kubernetes-monitoring/kubernetes-mixin" rel="noreferrer">kubernetes-mixin</a> project. There is an open issue (<a href="https://github.com/kubernetes-monitoring/kubernetes-mixin/issues/108" rel="noreferrer">#108</a>) to discuss this alert. I suggest that you read all the comments on this issue to better understand the problem.</p>
<p>In short, the problem is: When working with low CPU limits, spiky workloads can have low averages and still be being throttled.</p>
<p>Also, take a look at this issue (<a href="https://github.com/kubernetes/kubernetes/issues/67577" rel="noreferrer">#67577</a>) from Kubernetes project, which addresses a Kernel bug in CFS quotas that may cause unnecessary CPU throttling. The discussion is still open, and the Kubernetes project are even considering disabling CFS quotas for pods in the <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed" rel="noreferrer"><code>Guaranteed</code> QoS</a> (see <a href="https://github.com/kubernetes/kubernetes/issues/70585" rel="noreferrer">#70585</a> for reference).</p>
<p>Consider the following options:</p>
<ul>
<li>Increase (or even remove) your container CPU limits</li>
<li>Disable Kubernetes CFS quotas entirely
(kubelet's flag <code>--cpu-cfs-quota=false</code>)</li>
<li>Use a Kernel version that contains this fix (<a href="https://github.com/torvalds/linux/commit/512ac999d2755d2b7109e996a76b6fb8b888631d" rel="noreferrer">torvalds/linux 512ac99</a>)</li>
</ul>

CC BY-SA 4.0
2019-04-03T14:18:15.360Z
Thanks, it is nice to know that the issue is not on my installation. I don't want to increase or remove limits for now. And I am using AKS (a managed implementation of Kubernetes in Azure cloud) so I cannot disable CFS quotas. As per the comments in the issue, I am going to try to update the kernel version to see what happens.
jobou
CC BY-SA 4.0
2019-12-27T10:16:43.393Z
@jobou, have you update a kernel version? Was it helpful?
Vadim Yangunaev
CC BY-SA 4.0
2019-12-28T19:16:11.140Z
No sorry, I have not worked on this issue yet so I don't have any real feedback to give.
jobou
CC BY-SA 4.0
2020-12-14T21:33:42.603Z
We are still eeing this exact same thing in Dec 2020 and we are running on GKE 1.16.15.4300.  Our requests and limits are quite a bit higher than the ones in the question.  Removing CPU limits is not really an option for us.
Jon Carlson
CC BY-SA 4.0
2019-04-01T23:10:18.107Z
2021-05-26T15:46:59.860Z
2021-05-26T15:46:59.860Z
Eduardo Baitello
<p>I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling.</p>

<p>The Prometheus alert rules to identify this alert is :</p>

<pre><code>alert: CPUThrottlingHigh
expr: 100
  * sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_throttled_periods_total{container_name!=""}[5m]))
  / sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_periods_total[5m]))
  &gt; 25
for: 15m
</code></pre>

<p>If I look at one of the pods identified by this alert, it does not seem to have any reason to throttle :</p>

<pre><code>$ kubectl top pod -n monitoring my-pod
NAME            CPU(cores)   MEMORY(bytes)   
my-pod          0m           6Mi
</code></pre>

<p>This pod has one container with these resources setup :</p>

<pre><code>Limits:
  cpu:     100m
  memory:  128Mi
Requests:
  cpu:     25m
  memory:  64Mi
</code></pre>

<p>And the node that is hosting this pod is not under any heavy cpu use :</p>

<pre><code>$ kubectl -n monitoring top node aks-agentpool-node-1
NAME                       CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
aks-agentpool-node-1       853m         21%       11668Mi         101%
</code></pre>

<p>On grafana, if I look at the chart for this pod, it never goes above <code>0,000022</code> of cpu usage</p>

<p>Why is it throttling ?</p>

CC BY-SA 4.0
2019-03-14T08:06:09.957Z
Have you validated that you need to multiple by 100? Is it a decimal format for the percentage or an integer value?
sheldonhull
CC BY-SA 4.0
2019-03-14T11:15:16.427Z
I am not sure to understand. The expression `100m` is the same as `0.1` cpu.
jobou
CC BY-SA 4.0
2019-03-14T13:51:14.833Z
If that's the syntax for Prometheus then disregard. Was just checking.
sheldonhull
<docker><kubernetes><monitoring><prometheus>
Prometheus alert CPUThrottlingHigh raised but monitoring does not show it
jobou
# Prometheus alert CPUThrottlingHigh raised but monitoring does not show it

(Asked by: username_0 on 2019-02-27)

I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling.
The Prometheus alert rules to identify this alert is :
alert: CPUThrottlingHigh
expr: 100
  * sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_throttled_periods_total{container_name!=""}[5m]))
  / sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_periods_total[5m]))
  > 25
for: 15m

If I look at one of the pods identified by this alert, it does not seem to have any reason to throttle :
$ kubectl top pod -n monitoring my-pod
NAME            CPU(cores)   MEMORY(bytes)   
my-pod          0m           6Mi

This pod has one container with these resources setup :
Limits:
  cpu:     100m
  memory:  128Mi
Requests:
  cpu:     25m
  memory:  64Mi

And the node that is hosting this pod is not under any heavy cpu use :
$ kubectl -n monitoring top node aks-agentpool-node-1
NAME                       CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%   
aks-agentpool-node-1       853m         21%       11668Mi         101%

On grafana, if I look at the chart for this pod, it never goes above 0,000022 of cpu usage
Why is it throttling ?

**Tags:** docker, kubernetes, monitoring, prometheus

> **Comments:**
> **username_1**: Have you validated that you need to multiple by 100? Is it a decimal format for the percentage or an integer value?  
> **username_0**: I am not sure to understand. The expression `100m` is the same as `0.1` cpu.  
> **username_1**: If that's the syntax for Prometheus then disregard. Was just checking.  

---

## 2 answers

### Answer by username_4 on 2019-04-01. Score: 15 | Accepted answer

The `CPUThrottlingHigh` is an alert created by the [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin) project. There is an open issue ([#108](https://github.com/kubernetes-monitoring/kubernetes-mixin/issues/108)) to discuss this alert. I suggest that you read all the comments on this issue to better understand the problem.

In short, the problem is: When working with low CPU limits, spiky workloads can have low averages and still be being throttled.

Also, take a look at this issue ([#67577](https://github.com/kubernetes/kubernetes/issues/67577)) from Kubernetes project, which addresses a Kernel bug in CFS quotas that may cause unnecessary CPU throttling. The discussion is still open, and the Kubernetes project are even considering disabling CFS quotas for pods in the [`Guaranteed` QoS](https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/#create-a-pod-that-gets-assigned-a-qos-class-of-guaranteed) (see [#70585](https://github.com/kubernetes/kubernetes/issues/70585) for reference).

Consider the following options:

* Increase (or even remove) your container CPU limits
* Disable Kubernetes CFS quotas entirely
  (kubelet's flag `--cpu-cfs-quota=false`)
* Use a Kernel version that contains this fix ([torvalds/linux 512ac99](https://github.com/torvalds/linux/commit/512ac999d2755d2b7109e996a76b6fb8b888631d))

> **Comments:**
> **username_0**: Thanks, it is nice to know that the issue is not on my installation. I don't want to increase or remove limits for now. And I am using AKS (a managed implementation of Kubernetes in Azure cloud) so I cannot disable CFS quotas. As per the comments in the issue, I am going to try to update the kernel version to see what happens.  
> **username_5**: @username_0, have you update a kernel version? Was it helpful?  
> **username_0**: No sorry, I have not worked on this issue yet so I don't have any real feedback to give.  
> **username_6**: We are still eeing this exact same thing in Dec 2020 and we are running on GKE 1.16.15.4300.  Our requests and limits are quite a bit higher than the ones in the question.  Removing CPU limits is not really an option for us.  

---

### Answer by username_2 on 2021-02-08. Score: 2

To piggy back off of @eduardo-baitello answer, A third option is to increase the `CPUThrottlingPercent` config [here](https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/f7ca48cca5d9cadc9a2203b8c0b3bb3eb85f3294/config.libsonnet#L44)

> **Comments:**
> **username_3**: Mind posting this as a comment?  

---

