<p>You could raise the <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/" rel="nofollow noreferrer">scheduling priority</a> of your pod so that it will preempt (evict) the other pods if necessary.</p>
<p>You could also check that there isn’t a <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy" rel="nofollow noreferrer">restart policy</a> causing your pod to end (rather than restart its containers) when the processes terminate.</p>
<p>Normally though, those aren’t supposed to be issues <a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#scheduled-by-default-scheduler" rel="nofollow noreferrer">for daemonsets</a> anyway. You shouldn’t need to try to pre-reserve capacity (other than by ensuring the resource requests are ample for the contained processes). You may need to configure your daemonset to skip particular node types that have too little capacity for your collection of daemonsets.</p>
<p>(To reiterate, the most likely cause of the error described by the question is if one of your nodes only has 3GB total of memory, and there are two different daemonsets that request 2GB each of memory, which is unsatisfiable regardless of other pods.)</p>
<hr />
<p>Daemonsets are controlled quite differently from other pods. For a bare pod (or even a pod from a deployment) that you intend to assign to a particular node (say by setting <code>nodeName</code> or <code>nodeSelector</code> in the PodSpec, or merely by lack of a cluster autoscaler) then definitely do raise the scheduling priority (so that once created it will most likely be able to evict other pods if necessary to make space, notwithstanding several exceptions..). Also keep the default configuration of restarting any containers if they terminate (rather than relinquishing the requested memory back to the scheduler, then recreating a replacement pod..).</p>
<p>It simply is <strong>not</strong> a kubernetes pattern to reserve resources for a potential future pod. Or rather, you may think of a &quot;pod&quot; API object as already a pre-allocation for the node resources that could be consumed in future by <em>processes</em> in the pod's containers (but allocated before those containers even get created, and usually long before the processes will ever consume that amount of memory). Your pod could use an init container to wait arbitrarily after reserving node resources and before starting the main container process (and more complex orchestration might be achievable with a sidecar container configured to share process namespace..).</p>
<p>An exception is that placeholder pods are sometimes used to over-provision an autoscaled cluster. A pod with very low priority can still trigger the slow process of adding another node to the cluster, so that the new node has a chance to become ready <em>before</em> the scheduler needs it for a normal pod.</p>

CC BY-SA 4.0
2022-12-28T15:45:58.223Z
Is there a way to tell kubernetes reserve  some resources for a future create pod ?
chenxinlong
CC BY-SA 4.0
2022-12-29T08:55:34.887Z
@chenxinlong Why? (How is it that you already know what resources your pod will need, and why don't you wish to create the pod yet, and what is your objection to letting less-important pods try to use the resources in the meantime?)
benjimin
CC BY-SA 4.0
2022-12-29T10:19:16.317Z
I'm using kubevirt to run vm in pod(vm per pod), after vm shutdown kubevirt will delete this pod. During this vm is shutdown, other new create vm will occupied the shutdown vm's resources(cpu,memory,gpu...), which may causes the shutdown vm can not start successfully(because there's no more allocatable resources for 
 the pod of this shutdown vm).  So I was wondering how to reserve resources for these shutdown vms.
chenxinlong
CC BY-SA 4.0
2022-12-29T13:03:59.800Z
@chenxinlong Why do you want the pod on that specific node? What do you want to happen to the other pod? Is there some obstacle to adding more nodes?
benjimin
CC BY-SA 4.0
2022-12-29T17:20:27.793Z
adding more nodes should not even be a thought if the cluster autoscaling is configured properly. if cost is a concern there are so many other ways you can resolve this, you can use SPOT instances, move from intel machines to AMD which are 10 percent cheaper to run on AWS
Riv P
CC BY-SA 4.0
2022-12-30T05:52:04.777Z
Cuz there's also GPU resources in cluster,  let say the allocatable GPU number in cluster are fixed. After a vm with GPU shutdown, the vm pod deleted, other new create vm which requires GPU will occupie it. So the shutdown vm may not be able to start(vm pod start failed for insufficient GPU) which is not what I expected.
chenxinlong
CC BY-SA 4.0
2022-12-30T07:06:22.913Z
@chenxinlong you can use *taints and tolerations* to prevent the nodes with GPUs from being occupied by pods that don't need GPUs.
benjimin
CC BY-SA 4.0
2022-12-30T10:12:05.563Z
Actually other pods may need GPUs, what I expected is the shutdown vm will never start failed by insufficient resources. So that's why I'm looking a way to preserve resources for the shutdown vm.   But your points of priority and preempt gives me a clue, can I create a bare pod to preempt the pod of the shutdown vm, and when the vm restart, we preempt it back and we delete the bare pod ?
chenxinlong
CC BY-SA 4.0
2022-12-30T10:20:54.273Z
@chenxinlong I've already asked why you want the resources to be *idle* after your pod is deleted, rather than letting them be used in the interim by other pods (which you can automatically evict once your higher-priority pod is re-created). Could you perhaps create a new question that explains your actual problem in detail, since it seems to be unrelated to *this* question (which was about daemonsets)?
benjimin
CC BY-SA 4.0
2022-12-28T15:24:58.247Z
2022-12-29T08:56:28.193Z
2022-12-29T08:56:28.193Z
benjimin
<p>Let say there's a daemonset, each pod of this daemonset requests 1 CPU and 1 Gi memory. For some reason pod terminated, during this time kubernetes allocate cpu/memory resource to other pods which causes daemonset pod re-create failed(Insufficient cpu/memory resource).</p>
<p>Is there any way to reserve 1 CPU and 1 Gi memory for this pod during it terminated ?</p>

CC BY-SA 4.0
2022-12-28T16:19:30.633Z
Do you have some logs showing this error, and what else was on the same node at the time (and the total capacity of the node)?
benjimin
CC BY-SA 4.0
2022-12-29T04:53:07.773Z
@benjimin No, this is a assumption. Sorry for the misunderstanding,  I just want to know how to preserve some node resource for a future create pod.
chenxinlong
CC BY-SA 4.0
2022-12-29T17:18:25.623Z
if you have cluster autoscaling, the new pod will come up fine. Also if you have CPU and memory limits on the pods k8s cannot just allocate more cpu/memory to other pods. either you can remove the cpu/memory limits from your resources (there is a huge debate about this) or enable cluster autoscaling
Riv P
<kubernetes>
Reserve cpu and memory resources for pods in kubernetes?
chenxinlong
# Reserve cpu and memory resources for pods in kubernetes?

(Asked by: username_0 on 2022-12-28)

Let say there's a daemonset, each pod of this daemonset requests 1 CPU and 1 Gi memory. For some reason pod terminated, during this time kubernetes allocate cpu/memory resource to other pods which causes daemonset pod re-create failed(Insufficient cpu/memory resource).

Is there any way to reserve 1 CPU and 1 Gi memory for this pod during it terminated ?

**Tags:** kubernetes

> **Comments:**
> **username_1**: Do you have some logs showing this error, and what else was on the same node at the time (and the total capacity of the node)?  
> **username_0**: @username_1 No, this is a assumption. Sorry for the misunderstanding,  I just want to know how to preserve some node resource for a future create pod.  
> **username_2**: if you have cluster autoscaling, the new pod will come up fine. Also if you have CPU and memory limits on the pods k8s cannot just allocate more cpu/memory to other pods. either you can remove the cpu/memory limits from your resources (there is a huge debate about this) or enable cluster autoscaling  

---

## 1 answers

### Answer by username_1 on 2022-12-28. Score: 1

You could raise the [scheduling priority](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/) of your pod so that it will preempt (evict) the other pods if necessary.

You could also check that there isn’t a [restart policy](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy) causing your pod to end (rather than restart its containers) when the processes terminate.

Normally though, those aren’t supposed to be issues [for daemonsets](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#scheduled-by-default-scheduler) anyway. You shouldn’t need to try to pre-reserve capacity (other than by ensuring the resource requests are ample for the contained processes). You may need to configure your daemonset to skip particular node types that have too little capacity for your collection of daemonsets.

(To reiterate, the most likely cause of the error described by the question is if one of your nodes only has 3GB total of memory, and there are two different daemonsets that request 2GB each of memory, which is unsatisfiable regardless of other pods.)

---

Daemonsets are controlled quite differently from other pods. For a bare pod (or even a pod from a deployment) that you intend to assign to a particular node (say by setting `nodeName` or `nodeSelector` in the PodSpec, or merely by lack of a cluster autoscaler) then definitely do raise the scheduling priority (so that once created it will most likely be able to evict other pods if necessary to make space, notwithstanding several exceptions..). Also keep the default configuration of restarting any containers if they terminate (rather than relinquishing the requested memory back to the scheduler, then recreating a replacement pod..).

It simply is **not** a kubernetes pattern to reserve resources for a potential future pod. Or rather, you may think of a "pod" API object as already a pre-allocation for the node resources that could be consumed in future by *processes* in the pod's containers (but allocated before those containers even get created, and usually long before the processes will ever consume that amount of memory). Your pod could use an init container to wait arbitrarily after reserving node resources and before starting the main container process (and more complex orchestration might be achievable with a sidecar container configured to share process namespace..).

An exception is that placeholder pods are sometimes used to over-provision an autoscaled cluster. A pod with very low priority can still trigger the slow process of adding another node to the cluster, so that the new node has a chance to become ready *before* the scheduler needs it for a normal pod.

> **Comments:**
> **username_0**: Is there a way to tell kubernetes reserve  some resources for a future create pod ?  
> **username_1**: @username_0 Why? (How is it that you already know what resources your pod will need, and why don't you wish to create the pod yet, and what is your objection to letting less-important pods try to use the resources in the meantime?)  
> **username_0**: I'm using kubevirt to run vm in pod(vm per pod), after vm shutdown kubevirt will delete this pod. During this vm is shutdown, other new create vm will occupied the shutdown vm's resources(cpu,memory,gpu...), which may causes the shutdown vm can not start successfully(because there's no more allocatable resources for   the pod of this shutdown vm).  So I was wondering how to reserve resources for these shutdown vms.  
> **username_1**: @username_0 Why do you want the pod on that specific node? What do you want to happen to the other pod? Is there some obstacle to adding more nodes?  
> **username_2**: adding more nodes should not even be a thought if the cluster autoscaling is configured properly. if cost is a concern there are so many other ways you can resolve this, you can use SPOT instances, move from intel machines to AMD which are 10 percent cheaper to run on AWS  
> **username_0**: Cuz there's also GPU resources in cluster,  let say the allocatable GPU number in cluster are fixed. After a vm with GPU shutdown, the vm pod deleted, other new create vm which requires GPU will occupie it. So the shutdown vm may not be able to start(vm pod start failed for insufficient GPU) which is not what I expected.  
> **username_1**: @username_0 you can use *taints and tolerations* to prevent the nodes with GPUs from being occupied by pods that don't need GPUs.  
> **username_0**: Actually other pods may need GPUs, what I expected is the shutdown vm will never start failed by insufficient resources. So that's why I'm looking a way to preserve resources for the shutdown vm.   But your points of priority and preempt gives me a clue, can I create a bare pod to preempt the pod of the shutdown vm, and when the vm restart, we preempt it back and we delete the bare pod ?  
> **username_1**: @username_0 I've already asked why you want the resources to be *idle* after your pod is deleted, rather than letting them be used in the interim by other pods (which you can automatically evict once your higher-priority pod is re-created). Could you perhaps create a new question that explains your actual problem in detail, since it seems to be unrelated to *this* question (which was about daemonsets)?  

---

