<p>From the get-go, my educated guess would be memory leak your workloads. It seems that his happens pretty regular - every 12 hours, do you have some scheduled/cron job?</p>
<p>Alternative problem could be overcommitted resources.</p>
<p>First of all check <code>resources.limits</code> on workloads you are running on the EKS cluster. Make sure <code>resources.limits.memory</code> is set on every single container. You can play around with <a href="https://kubernetes.io/docs/concepts/policy/limit-range/" rel="nofollow noreferrer">limit ranges</a> to set defaults for a namespace; this way you &quot;protect&quot; your nodes to some extent (alternative is to use <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/" rel="nofollow noreferrer">resource quotas</a>.</p>
<p>I see you are using Datadog, you can leverage Kubernetes metrics to find spiking workload - <code>kubernetes.memory.working_set</code> (<a href="https://docs.datadoghq.com/containers/kubernetes/data_collected/" rel="nofollow noreferrer">reference</a>):</p>
<blockquote>
<p>Current working set in bytes - this is what the OOM killer is watching for</p>
</blockquote>
<p>The description of metric you are currently using (<a href="https://docs.datadoghq.com/metrics/guide/calculating-the-system-mem-used-metric/" rel="nofollow noreferrer">reference</a>):</p>
<blockquote>
<p>The <code>system.mem.usable</code> metric represents free memory plus cached memory plus buffers (on Linux, it reflects “MemAvailable” attribute from <code>/proc/meminfo</code> whenever possible).</p>
</blockquote>
<p>On a &quot;cluster&quot; side of things you may enable <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/" rel="nofollow noreferrer">&quot;resource reservation&quot;</a>, which may help with keeping system daemons (like <code>kubelet</code>) alive. Also read docs on node <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#hard-eviction-thresholds" rel="nofollow noreferrer">pressure evictions and thresholds</a>.</p>

CC BY-SA 4.0
2023-06-24T13:10:19.317Z
2023-06-24T13:10:19.317Z
Kyslik
<p>I was able to correlate the &quot;usable memory&quot; going down with events in kubernetes:</p>
<pre><code>Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 06 Jun 2023 19:31:34 +0000   Thu, 11 May 2023 15:24:32 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
</code></pre>
<p>The time of the <code>LastTransitionTime</code> coincides with the lowest memory and the transition to a lot of available memory.</p>
<p>Also, I think that during the times of low available memory I get a lot of OOM errors and processes are getting killed with exit code 137.</p>
<p>Is there something misconfigured in this Kubernetes cluster? I think the available memory should not ever go that low and it should start evicting pods earlier. Also I don't understand why the whole memory becomes available - this means that we are evicting ALL pods?</p>
<p><a href="https://i.sstatic.net/MamJK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MamJK.png" alt="memory available goes lower and lower then jumps up" /></a></p>

<kubernetes><eks>
EKS: Usable Memory goes lower and lower on the node to 5% then jumps up to 80% available
Costi
# EKS: Usable Memory goes lower and lower on the node to 5% then jumps up to 80% available

(Asked by: username_0 on 2023-06-06)

I was able to correlate the "usable memory" going down with events in kubernetes:
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 06 Jun 2023 19:31:34 +0000   Thu, 11 May 2023 15:24:32 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available

The time of the LastTransitionTime coincides with the lowest memory and the transition to a lot of available memory.
Also, I think that during the times of low available memory I get a lot of OOM errors and processes are getting killed with exit code 137.
Is there something misconfigured in this Kubernetes cluster? I think the available memory should not ever go that low and it should start evicting pods earlier. Also I don't understand why the whole memory becomes available - this means that we are evicting ALL pods?

**Tags:** kubernetes, eks


---

## 1 answers

### Answer by username_1 on 2023-06-24. Score: 0

From the get-go, my educated guess would be memory leak your workloads. It seems that his happens pretty regular - every 12 hours, do you have some scheduled/cron job?

Alternative problem could be overcommitted resources.

First of all check `resources.limits` on workloads you are running on the EKS cluster. Make sure `resources.limits.memory` is set on every single container. You can play around with [limit ranges](https://kubernetes.io/docs/concepts/policy/limit-range/) to set defaults for a namespace; this way you "protect" your nodes to some extent (alternative is to use [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/).

I see you are using Datadog, you can leverage Kubernetes metrics to find spiking workload - `kubernetes.memory.working_set` ([reference](https://docs.datadoghq.com/containers/kubernetes/data_collected/)):

> Current working set in bytes - this is what the OOM killer is watching for

The description of metric you are currently using ([reference](https://docs.datadoghq.com/metrics/guide/calculating-the-system-mem-used-metric/)):

> The `system.mem.usable` metric represents free memory plus cached memory plus buffers (on Linux, it reflects “MemAvailable” attribute from `/proc/meminfo` whenever possible).

On a "cluster" side of things you may enable ["resource reservation"](https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/), which may help with keeping system daemons (like `kubelet`) alive. Also read docs on node [pressure evictions and thresholds](https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#hard-eviction-thresholds).


---

