<p>This is because of the <a href="https://devops.stackexchange.com/a/16085/18965">default settings for the eviction manager (currently 5% hard, and 10% minimum for image reclaim)</a> safeguard a new pod exhausting the space on your hard disk, and the metrics are a bit overly conservative. You can change the eviction manager settings by providing a <code>--kubelet-arg</code> that overrides the <a href="https://devops.stackexchange.com/a/16085/18965">k3s defaults.</a>. This can be persisted by providing <a href="https://devops.stackexchange.com/q/16070/18965"><code>kubelet-arg</code> to your k3s <code>KUBECONFIG</code> at <code>/etc/rancher/k3s/config.yaml</code></a>.</p>
<p>Like this,</p>
<pre class="lang-yaml prettyprint-override"><code>kubelet-arg:
  - &quot;eviction-minimum-reclaim=imagefs.available=2%,nodefs.available=2%&quot;
  - &quot;eviction-hard=memory.available&lt;500Mi,nodefs.available&lt;10Gi&quot;
</code></pre>
<hr />
<p><a href="https://github.com/k3s-io/k3s/issues/5488#issuecomment-1147992468" rel="nofollow noreferrer">Note, while currently required, passing options by commandline to k3s is deprecated in upstream Kubernetes and is subject to removal in k3s when/if it gets finally removed.</a></p>

CC BY-SA 4.0
2022-06-06T18:15:31.517Z
2022-06-15T17:45:20.990Z
2022-06-15T17:45:20.990Z
Evan Carroll
<p>When I run <code>kubectl get pods -w --namespace kubeapps</code> I see all my pods are pending</p>
<pre><code>NAME                                                         READY   STATUS    RESTARTS   AGE
kubeapps-internal-dashboard-584c686cc6-qwjnf                 0/1     Pending   0          19h
kubeapps-585bc584db-j994c                                    0/1     Pending   0          19h
kubeapps-internal-apprepository-controller-5d7685c4f-zsxnk   0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-ddp8z                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-mzbtn              0/1     Pending   0          19h
kubeapps-internal-dashboard-584c686cc6-wxnfd                 0/1     Pending   0          19h
kubeapps-postgresql-0                                        0/1     Pending   0          19h
kubeapps-585bc584db-rgvfq                                    0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-tzjwf                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-rbdrm              0/1     Pending   0          19h
</code></pre>
<p>When I check <code>journalctl -u k3s</code>, I see</p>
<pre><code>E0604 11:37:07.861130    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik: object &quot;kube-system&quot;/&quot;chart-content&gt;
E0604 11:37:07.861184    1301 nestedpendingoperations.go:335] Operation for &quot;{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217&gt;
E0604 11:37:08.062023    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik-crd: object &quot;kube-system&quot;/&quot;chart-con&gt;
E0604 11:37:08.062042    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik: object &quot;kube-system&quot;/&quot;chart-values-t&gt;
E0604 11:37:08.062060    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik-crd: object &quot;kube-system&quot;/&quot;chart-valu&gt;
E0604 11:37:08.062089    1301 nestedpendingoperations.go:335] Operation for &quot;{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217&gt;
E0604 11:37:08.062099    1301 nestedpendingoperations.go:335] Operation for &quot;{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e&gt;
E0604 11:37:08.062105    1301 nestedpendingoperations.go:335] Operation for &quot;{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e&gt;
I0604 11:37:08.079010    1301 eviction_manager.go:338] &quot;Eviction manager: attempting to reclaim&quot; resourceName=&quot;ephemeral-storage&quot;
I0604 11:37:08.079027    1301 container_gc.go:85] &quot;Attempting to delete unused containers&quot;
I0604 11:37:08.079496    1301 image_gc_manager.go:327] &quot;Attempting to delete unused images&quot;
I0604 11:37:08.082310    1301 eviction_manager.go:349] &quot;Eviction manager: must evict pod(s) to reclaim&quot; resourceName=&quot;ephemeral-storage&quot;
I0604 11:37:08.082338    1301 eviction_manager.go:367] &quot;Eviction manager: pods ranked for eviction&quot; pods=[kube-system/coredns-d76bd69b-2svkn &gt;
E0604 11:37:08.082353    1301 eviction_manager.go:560] &quot;Eviction manager: cannot evict a critical pod&quot; pod=&quot;kube-system/coredns-d76bd69b-2svk&gt;
E0604 11:37:08.082359    1301 eviction_manager.go:560] &quot;Eviction manager: cannot evict a critical pod&quot; pod=&quot;kube-system/metrics-server-7cd5fc&gt;
E0604 11:37:08.082363    1301 eviction_manager.go:560] &quot;Eviction manager: cannot evict a critical pod&quot; pod=&quot;kube-system/local-path-provisione&gt;
I0604 11:37:08.082368    1301 eviction_manager.go:390] &quot;Eviction manager: unable to evict any pods from the node&quot;
</code></pre>
<p>I think this is a result of the eviction manager, because my system is <em>low</em> on disk space? Perhaps it's just looking at percent free, the below is the device from <code>df -h</code></p>
<p>I'm also seeing this in the error log,</p>
<pre><code>k3s[1301]: I0604 11:52:13.741674    1301 image_gc_manager.go:310] &quot;Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold&quot; usage=98 highThreshold=85 amountToFree=337223212236 lowThreshold=80
k3s[1301]: E0604 11:52:13.742880    1301 kubelet.go:1347] &quot;Image garbage collection failed multiple times in a row&quot; err=&quot;failed to garbage collect required amount of images. Wanted to free 337223212236 bytes, but freed 0 bytes&quot;
</code></pre>
<pre><code>Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/x1c7--vg-root  1.8T  1.7T   50G  98% /
</code></pre>
<p>Where are these settings on k3s?</p>

<rancher><k3s><kubelet>
k3s: Eviction manager: attempting to reclaim" resourceName="ephemeral-storage
Evan Carroll
# k3s: Eviction manager: attempting to reclaim" resourceName="ephemeral-storage

(Asked by: username_0 on 2022-06-04)

When I run kubectl get pods -w --namespace kubeapps I see all my pods are pending
NAME                                                         READY   STATUS    RESTARTS   AGE
kubeapps-internal-dashboard-584c686cc6-qwjnf                 0/1     Pending   0          19h
kubeapps-585bc584db-j994c                                    0/1     Pending   0          19h
kubeapps-internal-apprepository-controller-5d7685c4f-zsxnk   0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-ddp8z                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-mzbtn              0/1     Pending   0          19h
kubeapps-internal-dashboard-584c686cc6-wxnfd                 0/1     Pending   0          19h
kubeapps-postgresql-0                                        0/1     Pending   0          19h
kubeapps-585bc584db-rgvfq                                    0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-tzjwf                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-rbdrm              0/1     Pending   0          19h

When I check journalctl -u k3s, I see
E0604 11:37:07.861130    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik: object "kube-system"/"chart-content>
E0604 11:37:07.861184    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062023    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik-crd: object "kube-system"/"chart-con>
E0604 11:37:08.062042    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik: object "kube-system"/"chart-values-t>
E0604 11:37:08.062060    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik-crd: object "kube-system"/"chart-valu>
E0604 11:37:08.062089    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062099    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
E0604 11:37:08.062105    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
I0604 11:37:08.079010    1301 eviction_manager.go:338] "Eviction manager: attempting to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.079027    1301 container_gc.go:85] "Attempting to delete unused containers"
I0604 11:37:08.079496    1301 image_gc_manager.go:327] "Attempting to delete unused images"
I0604 11:37:08.082310    1301 eviction_manager.go:349] "Eviction manager: must evict pod(s) to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.082338    1301 eviction_manager.go:367] "Eviction manager: pods ranked for eviction" pods=[kube-system/coredns-d76bd69b-2svkn >
E0604 11:37:08.082353    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/coredns-d76bd69b-2svk>
E0604 11:37:08.082359    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/metrics-server-7cd5fc>
E0604 11:37:08.082363    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/local-path-provisione>
I0604 11:37:08.082368    1301 eviction_manager.go:390] "Eviction manager: unable to evict any pods from the node"

I think this is a result of the eviction manager, because my system is low on disk space? Perhaps it's just looking at percent free, the below is the device from df -h
I'm also seeing this in the error log,
k3s[1301]: I0604 11:52:13.741674    1301 image_gc_manager.go:310] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=98 highThreshold=85 amountToFree=337223212236 lowThreshold=80
k3s[1301]: E0604 11:52:13.742880    1301 kubelet.go:1347] "Image garbage collection failed multiple times in a row" err="failed to garbage collect required amount of images. Wanted to free 337223212236 bytes, but freed 0 bytes"

Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/x1c7--vg-root  1.8T  1.7T   50G  98% /

Where are these settings on k3s?

**Tags:** rancher, k3s, kubelet


---

## 1 answers

### Answer by username_0 on 2022-06-06. Score: 1 | Accepted answer

This is because of the default settings for the eviction manager (currently 5% hard, and 10% minimum for image reclaim) safeguard a new pod exhausting the space on your hard disk, and the metrics are a bit overly conservative. You can change the eviction manager settings by providing a --kubelet-arg that overrides the k3s defaults.. This can be persisted by providing kubelet-arg to your k3s KUBECONFIG at /etc/rancher/k3s/config.yaml.
Like this,
kubelet-arg:
  - "eviction-minimum-reclaim=imagefs.available=2%,nodefs.available=2%"
  - "eviction-hard=memory.available<500Mi,nodefs.available<10Gi"


Note, while currently required, passing options by commandline to k3s is deprecated in upstream Kubernetes and is subject to removal in k3s when/if it gets finally removed.


---

