<blockquote>
<p>3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules</p>
</blockquote>
<p>This means that ES trying to find a different node to deploy separately all the pods of ES. But the cause of your node count is not enough to run one pod on each node, the other pods remain pending state.</p>
<p>For more information read <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/" rel="nofollow noreferrer">here</a></p>
<p>So from here, you have a 2 choice Neo))</p>
<ol>
<li>Add more nodes until all pods will have been scheduled on the different nodes</li>
<li>Minimize your ES sts pod count to 3</li>
</ol>

CC BY-SA 4.0
2021-03-07T18:21:14.030Z
Is there not a way to have more than one es data pod running on each of the r5 instances? I specifically got the r5.2xl's so that I could run up to roughly 8 es data pods. Other general usage nodes that I am running can run more than one of any kind of pod if there is enough node resources. What makes these tainted nodes only able to run one es data pod at a time. It shouldn't be the nodes resources from what I'm seeing.
protobyte
CC BY-SA 4.0
2021-03-07T18:46:17.890Z
You can run more than one pod but it's highly not recommended and doesn't lie under the best practices of DevOps, mainly due to the high availability and fault tolerance.
Hakob
CC BY-SA 4.0
2021-03-07T19:02:14.147Z
Ok. That is fair enough. It's mostly trying to test right now to make sure that we can autoscale the pods up before scaling out the nodes. Also that is why there are three different r5 nodes also running es data pods. So one could go down and I'd still have 2 more. So aside from not recommended, and k8s doing this on other nodes/pods by default. Can you explain what it is in my setup I've described which is preventing it from scheduling more than one of the same pod per node?
protobyte
CC BY-SA 4.0
2021-03-07T19:06:34.487Z
https://devops.stackexchange.com/a/13497/10769

do you read the link above?
Hakob
CC BY-SA 4.0
2021-03-07T19:37:47.883Z
If I'm understanding it correctly. I need to change the `nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution` to `nodeSelector`?
protobyte
CC BY-SA 4.0
2021-03-07T19:40:41.820Z
Probably, it is. But again, I personally do not recommend doing it that way
Hakob
CC BY-SA 4.0
2021-03-08T20:44:44.150Z
Ok, finally had a chance to make this change. I can confirm that this is what was prohibiting me from scheduling more than one of the same pod to the same node. Thank you for your help and thank you for the best practices recommendation. It is more a clients request. I will stress this point to them once again that it is not recommended. Thank you.
protobyte
CC BY-SA 4.0
2021-03-08T20:55:07.567Z
I am excited to hear from you that my advice was to help you. You are welcome.
Hakob
CC BY-SA 4.0
2021-03-06T19:19:01.113Z
2021-03-07T19:09:09.560Z
2021-03-07T19:09:09.560Z
Hakob
<p>I have an elasticsearch cluster in a kubernetes cluster. I have the data pods going to memory optimized nodes which are tainted so that only the elasticsearch data pods get scheduled to the. Right now I have 3 memory optimized ec2 instances for these data pods. They are r5.2Xlarge's which have 64G of memory. Here is the output of one of these r5 nodes. (they all look the same)</p>
<pre><code>Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         8
  ephemeral-storage:           32461564Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      65049812Ki
  pods:                        110
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         8
  ephemeral-storage:           29916577333
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      64947412Ki
  pods:                        110
System Info:
  Machine ID:                 ec223b5ea23ea6bd5b06e8ed0a733d2d
  System UUID:                ec223b5e-a23e-a6bd-5b06-e8ed0a733d2d
  Boot ID:                    798aca5f-d9e1-4c9f-b75d-e16f7ba2d514
  Kernel Version:             5.4.0-1024-aws
  OS Image:                   Ubuntu 20.04.1 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.11
  Kubelet Version:            v1.18.10
  Kube-Proxy Version:         v1.18.10
Non-terminated Pods:          (5 in total)
  Namespace                   Name                                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                                        ------------  ----------  ---------------  -------------  ---
  amazon-cloudwatch           fluentd-cloudwatch-tzsv4                                    100m (1%)     0 (0%)      200Mi (0%)       400Mi (0%)     21d
  default                     prometheus-prometheus-node-exporter-tvmd4                   100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  es                          elasticsearch-data-0                                        500m (6%)     1 (12%)     8Gi (12%)        8Gi (12%)      14m
  kube-system                 calico-node-dhxg5                                           100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  kube-system                 kube-proxy-ip-10-1-12-115.us-gov-west-1.compute.internal    100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         900m (11%)    1 (12%)
  memory                      8392Mi (13%)  8592Mi (13%)
  ephemeral-storage           0 (0%)        0 (0%)
  hugepages-1Gi               0 (0%)        0 (0%)
  hugepages-2Mi               0 (0%)        0 (0%)
  attachable-volumes-aws-ebs  0             0
</code></pre>
<p>Here is what my cluster looks like</p>
<pre><code>kubectl get pods -n es
NAME                                                 READY   STATUS    RESTARTS   AGE
elasticsearch-client-0                               1/1     Running   0          77m
elasticsearch-client-1                               1/1     Running   0          77m
elasticsearch-data-0                                 1/1     Running   0          77m
elasticsearch-data-1                                 1/1     Running   0          77m
elasticsearch-data-2                                 1/1     Running   0          77m
elasticsearch-data-3                                 0/1     Pending   0          77m
elasticsearch-data-4                                 0/1     Pending   0          77m
elasticsearch-data-5                                 0/1     Pending   0          77m
elasticsearch-data-6                                 0/1     Pending   0          77m
elasticsearch-data-7                                 0/1     Pending   0          77m
elasticsearch-master-0                               2/2     Running   0          77m
elasticsearch-master-1                               2/2     Running   0          77m
prometheus-elasticsearch-exporter-6d6c5d49cf-4w7gc   1/1     Running   0          22h
</code></pre>
<p>Here is the events when I describe pod</p>
<pre><code>Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  56s (x5 over 3m35s)  default-scheduler  0/11 nodes are available: 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules, 5 node(s) didn't match node selector.
</code></pre>
<p>Here is my resource limits and requests for the data pods</p>
<pre><code>    Limits:
      cpu:     1
      memory:  8Gi
    Requests:
      cpu:      500m
      memory:   8Gi
</code></pre>
<p>Here is what my nodeAffinity looks like</p>
<pre><code>nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
      - key: es-data
        operator: In
        values:
        - &quot;true&quot;
</code></pre>
<p>And my tolerations</p>
<pre><code>tolerations:
  - key: &quot;es-data&quot;
    operator: &quot;Equal&quot;
    value: &quot;true&quot;
    effect: &quot;NoSchedule&quot;
</code></pre>
<p>And here is my node tolerance when I describe the node</p>
<pre><code>Taints:             es-data=true:NoSchedule
</code></pre>
<p>I tainted it like</p>
<pre><code>kubectl taint nodes &lt;node&gt; es-data=true:NoSchedule
</code></pre>
<p>According to my calculations based on my understand (which is probably wrong) my data pods are only asking for 8G of memory from a node which has 64G available, and only one pod requesting 8G of memory is already using it. So it should have theoretically 56G of memory left to other pods requesting to be scheduled to it. And even the memory being used shows me it's only 13% used. Why can't it schedule? How can I troubleshoot? Am I misunderstanding how this should work? What else can I tell you which helps troubleshoot this?</p>
<p>Resolution: Based on Hakob's comments, the issue is that I had <code>nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution</code> set, which is a hard requirement directing the scheduler to only schedule one to each node. What I needed to do in order to schedule more than one to each node was change it to <code>nodeSelector</code>. If you are reaching this, please note Hakob's recommendation for why it is not suggested best practice to do this. I agree with that advice. Though in my case it is a requirement coming from client and did not have the option even after have discussion why they should not be doing this. So please take that into consideration when applying this change.</p>

CC BY-SA 4.0
2021-03-06T10:20:20.850Z
Probably the anti-affinity rule trigger the memory rule along it. I'd first work on the affinity problem first.
Tensibai
CC BY-SA 4.0
2021-03-06T16:56:43.613Z
Could you elaborate please. The affinity should be ok, since the first three are getting scheduled to the appropriate memory optimized tainted nodes just fine. Shouldn't they? And they never get scheduled where they are not suppose to. No other pods are scheduled to these memory optimized nodes. So everything seems just as I expected, until more than 1 of these pods try to get scheduled to one of the memory optimized.
protobyte
CC BY-SA 4.0
2021-03-07T23:35:13.040Z
Roughly what is in the answer: you ask for your pods to NOT be on the same node, so obviously it can only start as much pods as you have nodes.
Tensibai
<kubernetes>
Why is there insufficient memory on kubernetes node
protobyte
# Why is there insufficient memory on kubernetes node

(Asked by: username_0 on 2021-03-04)

I have an elasticsearch cluster in a kubernetes cluster. I have the data pods going to memory optimized nodes which are tainted so that only the elasticsearch data pods get scheduled to the. Right now I have 3 memory optimized ec2 instances for these data pods. They are r5.2Xlarge's which have 64G of memory. Here is the output of one of these r5 nodes. (they all look the same)
Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         8
  ephemeral-storage:           32461564Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      65049812Ki
  pods:                        110
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         8
  ephemeral-storage:           29916577333
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      64947412Ki
  pods:                        110
System Info:
  Machine ID:                 ec223b5ea23ea6bd5b06e8ed0a733d2d
  System UUID:                ec223b5e-a23e-a6bd-5b06-e8ed0a733d2d
  Boot ID:                    798aca5f-d9e1-4c9f-b75d-e16f7ba2d514
  Kernel Version:             5.4.0-1024-aws
  OS Image:                   Ubuntu 20.04.1 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.11
  Kubelet Version:            v1.18.10
  Kube-Proxy Version:         v1.18.10
Non-terminated Pods:          (5 in total)
  Namespace                   Name                                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                                        ------------  ----------  ---------------  -------------  ---
  amazon-cloudwatch           fluentd-cloudwatch-tzsv4                                    100m (1%)     0 (0%)      200Mi (0%)       400Mi (0%)     21d
  default                     prometheus-prometheus-node-exporter-tvmd4                   100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  es                          elasticsearch-data-0                                        500m (6%)     1 (12%)     8Gi (12%)        8Gi (12%)      14m
  kube-system                 calico-node-dhxg5                                           100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  kube-system                 kube-proxy-ip-10-1-12-115.us-gov-west-1.compute.internal    100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         900m (11%)    1 (12%)
  memory                      8392Mi (13%)  8592Mi (13%)
  ephemeral-storage           0 (0%)        0 (0%)
  hugepages-1Gi               0 (0%)        0 (0%)
  hugepages-2Mi               0 (0%)        0 (0%)
  attachable-volumes-aws-ebs  0             0

Here is what my cluster looks like
kubectl get pods -n es
NAME                                                 READY   STATUS    RESTARTS   AGE
elasticsearch-client-0                               1/1     Running   0          77m
elasticsearch-client-1                               1/1     Running   0          77m
elasticsearch-data-0                                 1/1     Running   0          77m
elasticsearch-data-1                                 1/1     Running   0          77m
elasticsearch-data-2                                 1/1     Running   0          77m
elasticsearch-data-3                                 0/1     Pending   0          77m
elasticsearch-data-4                                 0/1     Pending   0          77m
elasticsearch-data-5                                 0/1     Pending   0          77m
elasticsearch-data-6                                 0/1     Pending   0          77m
elasticsearch-data-7                                 0/1     Pending   0          77m
elasticsearch-master-0                               2/2     Running   0          77m
elasticsearch-master-1                               2/2     Running   0          77m
prometheus-elasticsearch-exporter-6d6c5d49cf-4w7gc   1/1     Running   0          22h

Here is the events when I describe pod
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  56s (x5 over 3m35s)  default-scheduler  0/11 nodes are available: 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules, 5 node(s) didn't match node selector.

Here is my resource limits and requests for the data pods
    Limits:
      cpu:     1
      memory:  8Gi
    Requests:
      cpu:      500m
      memory:   8Gi

Here is what my nodeAffinity looks like
nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
      - key: es-data
        operator: In
        values:
        - "true"

And my tolerations
tolerations:
  - key: "es-data"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

And here is my node tolerance when I describe the node
Taints:             es-data=true:NoSchedule

I tainted it like
kubectl taint nodes  es-data=true:NoSchedule

According to my calculations based on my understand (which is probably wrong) my data pods are only asking for 8G of memory from a node which has 64G available, and only one pod requesting 8G of memory is already using it. So it should have theoretically 56G of memory left to other pods requesting to be scheduled to it. And even the memory being used shows me it's only 13% used. Why can't it schedule? How can I troubleshoot? Am I misunderstanding how this should work? What else can I tell you which helps troubleshoot this?
Resolution: Based on username_2's comments, the issue is that I had nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution set, which is a hard requirement directing the scheduler to only schedule one to each node. What I needed to do in order to schedule more than one to each node was change it to nodeSelector. If you are reaching this, please note username_2's recommendation for why it is not suggested best practice to do this. I agree with that advice. Though in my case it is a requirement coming from client and did not have the option even after have discussion why they should not be doing this. So please take that into consideration when applying this change.

**Tags:** kubernetes

> **Comments:**
> **username_1**: Probably the anti-affinity rule trigger the memory rule along it. I'd first work on the affinity problem first.  
> **username_0**: Could you elaborate please. The affinity should be ok, since the first three are getting scheduled to the appropriate memory optimized tainted nodes just fine. Shouldn't they? And they never get scheduled where they are not suppose to. No other pods are scheduled to these memory optimized nodes. So everything seems just as I expected, until more than 1 of these pods try to get scheduled to one of the memory optimized.  
> **username_1**: Roughly what is in the answer: you ask for your pods to NOT be on the same node, so obviously it can only start as much pods as you have nodes.  

---

## 1 answers

### Answer by username_2 on 2021-03-06. Score: 2 | Accepted answer

> 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules

This means that ES trying to find a different node to deploy separately all the pods of ES. But the cause of your node count is not enough to run one pod on each node, the other pods remain pending state.

For more information read [here](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/)

So from here, you have a 2 choice Neo))

1. Add more nodes until all pods will have been scheduled on the different nodes
2. Minimize your ES sts pod count to 3

> **Comments:**
> **username_0**: Is there not a way to have more than one es data pod running on each of the r5 instances? I specifically got the r5.2xl's so that I could run up to roughly 8 es data pods. Other general usage nodes that I am running can run more than one of any kind of pod if there is enough node resources. What makes these tainted nodes only able to run one es data pod at a time. It shouldn't be the nodes resources from what I'm seeing.  
> **username_2**: You can run more than one pod but it's highly not recommended and doesn't lie under the best practices of DevOps, mainly due to the high availability and fault tolerance.  
> **username_0**: Ok. That is fair enough. It's mostly trying to test right now to make sure that we can autoscale the pods up before scaling out the nodes. Also that is why there are three different r5 nodes also running es data pods. So one could go down and I'd still have 2 more. So aside from not recommended, and k8s doing this on other nodes/pods by default. Can you explain what it is in my setup I've described which is preventing it from scheduling more than one of the same pod per node?  
> **username_2**: https://devops.stackexchange.com/a/13497/10769  do you read the link above?  
> **username_0**: If I'm understanding it correctly. I need to change the `nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution` to `nodeSelector`?  
> **username_2**: Probably, it is. But again, I personally do not recommend doing it that way  
> **username_0**: Ok, finally had a chance to make this change. I can confirm that this is what was prohibiting me from scheduling more than one of the same pod to the same node. Thank you for your help and thank you for the best practices recommendation. It is more a clients request. I will stress this point to them once again that it is not recommended. Thank you.  
> **username_2**: I am excited to hear from you that my advice was to help you. You are welcome.  

---

