<p>I would first of all question the need for this, to understand exactly why you think this is necessary.</p>
<p>If you understand how your the application works and how it consumes memory over time, you can predict how much RAM it will need. Let's say, for example, you know that your application will eventually use more than 32GB RAM over time, and you want to make sure that future pods will be scheduled in other nodes that have at least that amount of memory available.</p>
<p>If that is the case, this seems like a resource request/limits adjustment in the Pod spec. When you define <code>.resources.requests</code> for the container, you are informing Kubernetes the <strong>minimum amount of memory it needs to start</strong>. When the kube-scheduler decides where the Pod will run, it will take that into consideration. It will sum up all <code>.resources.requests</code> from the Pods already running in the node and subtract that from the total amount of RAM the node has. It the remaining value is enough to accommodate a new Pod's <code>.resources.requests</code>, then the new Pod can be scheduled into that node.</p>
<p>This logic does not consider how much RAM the containers are actually using during runtime, but you can use that to control how many pods are allowed into each node, by controlling how many can be &quot;crammed&quot; into it at one time. On some dev/staging environments, for example, we set very low <code>requests</code> values for all containers because we want to oversubscribe the cluster. Most of the time, that environment is under no pressure and is only used for testing and development, so we can get away with cramming as many pods as possible into every node. For production environments, we set realistic <code>requests</code> values to meet the application's expectation.</p>
<p>You can think of <code>.resources.requests</code> as a &quot;reservation&quot; to make sure that containers will have the minimum expected amount of resources they need to run. As an example in your case, if you se <code>requests.memory</code> to 35GB, you can guarantee that each pod will be scheduled only in the nodes where that amount of memory has still not reserved for the current Pods in that node. That way, you can guarantee that your pod will have at least that much memory to begin with, and other instances of the same Deployment will be forced to be scheduled into other nodes.</p>
<p>If you wish to really control the scheduling logic using real time metrics, and only reschedule pods after they reach 32GB during runtime, then you will need to write your custom scheduler and replace the native <code>kube-scheduler</code> that Kubernetes ships with.</p>

CC BY-SA 4.0
2024-12-29T23:32:48.607Z
2024-12-29T23:33:14.203Z
2024-12-29T23:33:14.203Z
Julio Henrique Morimoto
<p>Use a custom scheduler or a scheduler extender. First, ensure the Kubernetes Metrics Server is running to collect real-time memory data from the nodes. Then, either modify the default scheduler / create a custom one that checks the node’s memory usage before placing new pods.
If the memory usage is over 50%, the scheduler should block new pods from being scheduled, even if there are still available resources based on the pod's requests.
Also set up Prometheus to monitor memory usage and trigger alerts or actions when a node hits the threshold, preventing new pod placements. This approach should ig.</p>

CC BY-SA 4.0
2024-12-15T17:04:29.587Z
well said , but what if node memory usage is less than 50% and our scheduler schedules a pod on it which increases the node's memory utilisation above 50%. what to do in this case?
Swetank Rawat
CC BY-SA 4.0
2024-12-14T17:24:12.317Z
2024-12-14T17:24:48.897Z
2024-12-14T17:24:48.897Z
Kiyomizu
<p>to achieve above scenario :
there is a flag -&gt; --eviction-hard=memory.available=1000Mi(for eg).
by passing this flag in kubelet extra argument, during startup of kubelet , changes can be seen in kubelet logs too.</p>

CC BY-SA 4.0
2024-12-18T14:35:29.077Z
2024-12-18T14:35:29.077Z
Swetank Rawat
<p>For eg: worker node(W) have 64GB of memory and 16vCPUs</p>
<p>resources assigned to a pod : cpu - 4 , memory- 8GB</p>
<p>Now only 4 pods can be scheduled on worker node(W).</p>
<p>How can i achieve this ?</p>
<p>Node affinity , node selectors or taint and tolerations do not work in this case.</p>

<kubernetes><amazon-web-services><containers><eks><autoscaling>
In kubernetes, I want to restrict my pod scheduling on worker node if memory utilisation of worker node goes above 50% of its capacity?
Swetank Rawat
# In kubernetes, I want to restrict my pod scheduling on worker node if memory utilisation of worker node goes above 50% of its capacity?

(Asked by: username_0 on 2024-12-14)

For eg: worker node(W) have 64GB of memory and 16vCPUs

resources assigned to a pod : cpu - 4 , memory- 8GB

Now only 4 pods can be scheduled on worker node(W).

How can i achieve this ?

Node affinity , node selectors or taint and tolerations do not work in this case.

**Tags:** kubernetes, amazon-web-services, containers, eks, autoscaling


---

## 3 answers

### Answer by username_1 on 2024-12-29. Score: 0

I would first of all question the need for this, to understand exactly why you think this is necessary.

If you understand how your the application works and how it consumes memory over time, you can predict how much RAM it will need. Let's say, for example, you know that your application will eventually use more than 32GB RAM over time, and you want to make sure that future pods will be scheduled in other nodes that have at least that amount of memory available.

If that is the case, this seems like a resource request/limits adjustment in the Pod spec. When you define `.resources.requests` for the container, you are informing Kubernetes the **minimum amount of memory it needs to start**. When the kube-scheduler decides where the Pod will run, it will take that into consideration. It will sum up all `.resources.requests` from the Pods already running in the node and subtract that from the total amount of RAM the node has. It the remaining value is enough to accommodate a new Pod's `.resources.requests`, then the new Pod can be scheduled into that node.

This logic does not consider how much RAM the containers are actually using during runtime, but you can use that to control how many pods are allowed into each node, by controlling how many can be "crammed" into it at one time. On some dev/staging environments, for example, we set very low `requests` values for all containers because we want to oversubscribe the cluster. Most of the time, that environment is under no pressure and is only used for testing and development, so we can get away with cramming as many pods as possible into every node. For production environments, we set realistic `requests` values to meet the application's expectation.

You can think of `.resources.requests` as a "reservation" to make sure that containers will have the minimum expected amount of resources they need to run. As an example in your case, if you se `requests.memory` to 35GB, you can guarantee that each pod will be scheduled only in the nodes where that amount of memory has still not reserved for the current Pods in that node. That way, you can guarantee that your pod will have at least that much memory to begin with, and other instances of the same Deployment will be forced to be scheduled into other nodes.

If you wish to really control the scheduling logic using real time metrics, and only reschedule pods after they reach 32GB during runtime, then you will need to write your custom scheduler and replace the native `kube-scheduler` that Kubernetes ships with.


---

### Answer by username_2 on 2024-12-14. Score: 0

Use a custom scheduler or a scheduler extender. First, ensure the Kubernetes Metrics Server is running to collect real-time memory data from the nodes. Then, either modify the default scheduler / create a custom one that checks the node’s memory usage before placing new pods.
If the memory usage is over 50%, the scheduler should block new pods from being scheduled, even if there are still available resources based on the pod's requests.
Also set up Prometheus to monitor memory usage and trigger alerts or actions when a node hits the threshold, preventing new pod placements. This approach should ig.

> **Comments:**
> **username_0**: well said , but what if node memory usage is less than 50% and our scheduler schedules a pod on it which increases the node's memory utilisation above 50%. what to do in this case?  

---

### Answer by username_0 on 2024-12-18. Score: 0

to achieve above scenario :
there is a flag -> --eviction-hard=memory.available=1000Mi(for eg).
by passing this flag in kubelet extra argument, during startup of kubelet , changes can be seen in kubelet logs too.


---

