<p>I had a node get automatically removed from our kubernetes cluster by the autoscaler.  However, it appears the scheduler didn't get the memo, and it's still trying to assign pods to the dead node:</p>

<pre><code>I0508 18:42:38.615641       7 scheduler.go:166] Failed to schedule pod: default/auth--web-1859671213-t95st
E0508 18:42:38.615718       7 factory.go:659] Error scheduling default auth--web-1859671213-t95st: node 'ip-10-100-70-212.ec2.internal' not found; retrying
I0508 18:42:38.615778       7 factory.go:734] Updating pod condition for default/auth--web-1859671213-t95st to (PodScheduled==False)
I0508 18:42:38.615915       7 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"auth--web-1859671213-t95st", UID:"8f42739e-71b3-11e9-b6cb-06d5afd487fe", APIVersion:"v1", ResourceVersion:"183075022", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' node 'ip-10-100-70-212.ec2.internal' not found
</code></pre>

<p>A <code>kubectl get nodes</code> confirms that the node is no longer a member of the cluster.  I can see the event in the AWS auto scaling group logs where the node was removed.  It's definitely dead and gone.</p>

<p>I found a series of bug reports where this happened while leaving the scheduler in bad state, culminating in a PR to fix the problem:</p>

<ul>
<li><a href="https://github.com/kubernetes/kubernetes/issues/43806" rel="nofollow noreferrer">kube-scheduler gets stuck if there's a pod in a namespace that doesn't exist assigned to a node that doesn't exist</a></li>
<li><a href="https://github.com/kubernetes/kubernetes/issues/56057" rel="nofollow noreferrer">Pending pods with affinity rules looking for non-existing node</a></li>
<li><a href="https://github.com/kubernetes/kubernetes/issues/56261" rel="nofollow noreferrer">Scheduler should delete a node from its cache if it gets "node not found" error</a></li>
<li>Bug fix, <a href="https://github.com/kubernetes/kubernetes/pull/56622" rel="nofollow noreferrer">Delete a node from its cache if it gets node not found error</a></li>
</ul>

<p>However, the version of kubernetes we're running is old (1.7), so we're currently missing out on the patch.  Clearly, I should upgrade the cluster to a modern version.</p>

<p>But I'd still like to get the Pending pods back up and running first.  Is there a way I can manually force the scheduler to update its cache, or to otherwise force the pods to be rescheduled onto a node that actually exists?</p>

<p>Things I've already tried:</p>

<ul>
<li>Adding more nodes to the cluster, to verify that there are enough resources to run the pods.</li>
<li>Restarting the scheduler, it comes back online with the same error.</li>
<li>Deleting the Pending pod, the replacement just tries to schedule back onto the same dead node.</li>
</ul>

CC BY-SA 4.0
2019-05-20T20:24:03.053Z
We've abandoned this cluster for a new one running a modern version of kubernetes.  The old cluster is shut down, it's unlikely we'll ever be able to debug this issue further again.
Brad Koch
CC BY-SA 4.0
2019-12-25T09:41:58.807Z
Thanks for the update. I assume that the reported issue does not appear in the new cluster?
030
CC BY-SA 4.0
2019-12-25T21:05:51.660Z
Yeah, issue has never shown up again, likely because the new cluster is running code with the mentioned patch.  No point trying to repro this, any reasonably recent build should be unaffected.
Brad Koch
<kubernetes>
Why is the Kubernetes scheduler trying to assign pods to nodes that no longer exist?
Brad Koch
# Why is the Kubernetes scheduler trying to assign pods to nodes that no longer exist?

(Asked by: username_0 on 2019-05-08)

I had a node get automatically removed from our kubernetes cluster by the autoscaler.  However, it appears the scheduler didn't get the memo, and it's still trying to assign pods to the dead node:
I0508 18:42:38.615641       7 scheduler.go:166] Failed to schedule pod: default/auth--web-1859671213-t95st
E0508 18:42:38.615718       7 factory.go:659] Error scheduling default auth--web-1859671213-t95st: node 'ip-10-100-70-212.ec2.internal' not found; retrying
I0508 18:42:38.615778       7 factory.go:734] Updating pod condition for default/auth--web-1859671213-t95st to (PodScheduled==False)
I0508 18:42:38.615915       7 event.go:218] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"auth--web-1859671213-t95st", UID:"8f42739e-71b3-11e9-b6cb-06d5afd487fe", APIVersion:"v1", ResourceVersion:"183075022", FieldPath:""}): type: 'Warning' reason: 'FailedScheduling' node 'ip-10-100-70-212.ec2.internal' not found

A kubectl get nodes confirms that the node is no longer a member of the cluster.  I can see the event in the AWS auto scaling group logs where the node was removed.  It's definitely dead and gone.
I found a series of bug reports where this happened while leaving the scheduler in bad state, culminating in a PR to fix the problem:

kube-scheduler gets stuck if there's a pod in a namespace that doesn't exist assigned to a node that doesn't exist
Pending pods with affinity rules looking for non-existing node
Scheduler should delete a node from its cache if it gets "node not found" error
Bug fix, Delete a node from its cache if it gets node not found error

However, the version of kubernetes we're running is old (1.7), so we're currently missing out on the patch.  Clearly, I should upgrade the cluster to a modern version.
But I'd still like to get the Pending pods back up and running first.  Is there a way I can manually force the scheduler to update its cache, or to otherwise force the pods to be rescheduled onto a node that actually exists?
Things I've already tried:

Adding more nodes to the cluster, to verify that there are enough resources to run the pods.
Restarting the scheduler, it comes back online with the same error.
Deleting the Pending pod, the replacement just tries to schedule back onto the same dead node.

**Tags:** kubernetes

> **Comments:**
> **username_0**: We've abandoned this cluster for a new one running a modern version of kubernetes.  The old cluster is shut down, it's unlikely we'll ever be able to debug this issue further again.  
> **username_1**: Thanks for the update. I assume that the reported issue does not appear in the new cluster?  
> **username_0**: Yeah, issue has never shown up again, likely because the new cluster is running code with the mentioned patch.  No point trying to repro this, any reasonably recent build should be unaffected.  

---

## 0 answers

