<p>My saviour was</p>
<p><code>kubectl edit deploy portainer -n portainer -o yaml</code></p>
<p>changed the node name, saved the file and watched everything work</p>

CC BY-SA 4.0
2023-10-29T11:28:27.307Z
2023-10-29T11:28:27.307Z
JoSSte
<p>I have a home lab running k3s.</p>
<blockquote>
<pre><code>NAME         STATUS   ROLES                         AGE    VERSION
k8s-node-001 Ready    control-plane,master,worker   302d   v1.27.6+k3s1
k8s-node-002 Ready    worker                        302d   v1.27.6+k3s1
</code></pre>
</blockquote>
<p>I recently removed a node forcefully, since the disk stopped working.
(k8s-node-003 was the failure - now deleted from the cluster)
Unfortunately that node was running the pvc for my portainer setup.</p>
<p>I looked for a way to &quot;move&quot; the pvc to another node without much luck. So I deleted it <code>kubectl delete pvc portainer -n portainer</code></p>
<p>I scaled down the portainer deployment with <code>kubectl scale deployment portainer -n portainer --replicas=0</code> and tried to scale it back up <code>kubectl scale deployment portainer -n portainer --replicas=1</code></p>
<p>Inspecting the pod under deployment <code>kubectl describe pod portainer-858644df85-sjdbk -n portainer</code> I see that</p>
<blockquote>
<pre><code>...

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  95s   default-scheduler  0/2 nodes are available: persistentvolumeclaim &quot;portainer&quot; not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  13s   default-scheduler  0/2 nodes are available: persistentvolumeclaim &quot;portainer&quot; not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
</code></pre>
</blockquote>
<p>I verified that the claim is gone</p>
<p><code>kubectl get pv -n portainer portainer</code></p>
<blockquote>
<pre><code>Error from server (NotFound): persistentvolumes &quot;portainer&quot; not found
</code></pre>
</blockquote>
<p><code>kubectl get pvc -n portainer portainer</code></p>
<blockquote>
<pre><code>Error from server (NotFound): persistentvolumeclaims &quot;portainer&quot; not found
</code></pre>
</blockquote>
<p>I guess I should have saved the pvc and then reinstated that using a command like <code>kubectl get pvc --namespace portainer -o json &gt; portainer-pvc.yaml</code> and then reinstated it with something like <code>kubectl apply -f portainer-pvc.yaml -n portainer</code> but 20/20 hindsight is not useful now...</p>
<p>Is it sufficient to find the portainer pvc definition and applying that?</p>
<h2>UPDATE</h2>
<p>I found the command i used to update portainer and ran that</p>
<pre class="lang-sh prettyprint-override"><code>helm upgrade --install --create-namespace -n portainer portainer portainer/portainer \
     --set enterpriseEdition.enabled=true \
     --set tls.force=true \
     --set tls.existingSecret=portainer-tls-secret
</code></pre>
<p>which appears to have reinitialised the pvc</p>
<p><code>kubectl get pvc -n portainer portainer</code></p>
<blockquote>
<pre><code>NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
portainer   Pending                                      local-path     4m12s
</code></pre>
</blockquote>
<p>...but it seems that there is some other reference to the defunct node:</p>
<blockquote>
<pre><code>Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  4m24s  default-scheduler  0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
</code></pre>
</blockquote>
<p>Is there a way to alter the configuration besides wiping the whole portainer configuration and reaapplying it ?</p>

<kubernetes><kubectl><k3s><persistent-volumes>
Redefine deployment (Portainer) pvc after node failure
JoSSte
# Redefine deployment (Portainer) pvc after node failure

(Asked by: username_0 on 2023-10-28)

I have a home lab running k3s.

NAME         STATUS   ROLES                         AGE    VERSION
k8s-node-001 Ready    control-plane,master,worker   302d   v1.27.6+k3s1
k8s-node-002 Ready    worker                        302d   v1.27.6+k3s1


I recently removed a node forcefully, since the disk stopped working.
(k8s-node-003 was the failure - now deleted from the cluster)
Unfortunately that node was running the pvc for my portainer setup.
I looked for a way to "move" the pvc to another node without much luck. So I deleted it kubectl delete pvc portainer -n portainer
I scaled down the portainer deployment with kubectl scale deployment portainer -n portainer --replicas=0 and tried to scale it back up kubectl scale deployment portainer -n portainer --replicas=1
Inspecting the pod under deployment kubectl describe pod portainer-858644df85-sjdbk -n portainer I see that

...

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  95s   default-scheduler  0/2 nodes are available: persistentvolumeclaim "portainer" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
  Warning  FailedScheduling  13s   default-scheduler  0/2 nodes are available: persistentvolumeclaim "portainer" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..


I verified that the claim is gone
kubectl get pv -n portainer portainer

Error from server (NotFound): persistentvolumes "portainer" not found


kubectl get pvc -n portainer portainer

Error from server (NotFound): persistentvolumeclaims "portainer" not found


I guess I should have saved the pvc and then reinstated that using a command like kubectl get pvc --namespace portainer -o json > portainer-pvc.yaml and then reinstated it with something like kubectl apply -f portainer-pvc.yaml -n portainer but 20/20 hindsight is not useful now...
Is it sufficient to find the portainer pvc definition and applying that?
UPDATE
I found the command i used to update portainer and ran that
helm upgrade --install --create-namespace -n portainer portainer portainer/portainer \
     --set enterpriseEdition.enabled=true \
     --set tls.force=true \
     --set tls.existingSecret=portainer-tls-secret

which appears to have reinitialised the pvc
kubectl get pvc -n portainer portainer

NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
portainer   Pending                                      local-path     4m12s


...but it seems that there is some other reference to the defunct node:

Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  4m24s  default-scheduler  0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..


Is there a way to alter the configuration besides wiping the whole portainer configuration and reaapplying it ?

**Tags:** kubernetes, kubectl, k3s, persistent-volumes


---

## 1 answers

### Answer by username_0 on 2023-10-29. Score: 0 | Accepted answer

My saviour was

`kubectl edit deploy portainer -n portainer -o yaml`

changed the node name, saved the file and watched everything work


---

