<p>I resolved this by using a different helm chart. I believe my original problem was caused by a version incompatibility.</p>
<pre><code>resource &quot;helm_release&quot; &quot;metrics_server&quot; {
  name       = &quot;metrics-server&quot;
  repository = &quot;https://kubernetes-sigs.github.io/metrics-server&quot;
  chart      = &quot;metrics-server&quot;
  namespace  = &quot;kube-system&quot;
}
</code></pre>

CC BY-SA 4.0
2024-01-17T12:11:09.197Z
2024-01-17T12:11:09.197Z
Don Draper
<p>I am attempting to setup metrics-server to server my 'horizon pod autoscaling' and manage my app pods scaling based on CPU. I am unable to get the metrics-server to run properly which fails with <strong>Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve</strong>. You can tell I am using EKS and managing the cluster using Terraform. Can anyone offer any insights?</p>
<pre><code>Versions
  - EKS 1.27
  - Helm 3.12.0
</code></pre>
<p>This is my terraform script showing this specific piece of my cluster:</p>
<pre><code>resource &quot;kubernetes_namespace&quot; &quot;metric-server&quot; {
  metadata {
    name = &quot;${var.cluster_name}-metric-server&quot;
  }
}

resource &quot;helm_release&quot; &quot;metric_server&quot; {
  name       = &quot;metric-server&quot;
  repository = &quot;https://charts.bitnami.com/bitnami&quot;
  chart      = &quot;metric-server&quot;
  namespace  = &quot;${var.cluster_name}-metric-server&quot;
  
  values = [
    &quot;${file(&quot;./values/metrics-server.yaml&quot;)}&quot;
  ]
}
</code></pre>
<p>The contents of ./values/metrics-server.yaml as follows.</p>
<pre><code>spec:
  containers:
  - arg:
      - --metric-resolution=18s
</code></pre>
<p>This is my cluster (kube-system)</p>
<pre><code>kubectl get po -n kube-system
NAME                                  READY   STATUS    RESTARTS       AGE
NAMESPACE                 NAME                                     READY   STATUS      RESTARTS        AGE
default                   reloader-reloader-7568f766fb-mkdzh       1/1     Running     0               3d8h
kube-system               aws-node-8w7lc                           2/2     Running     0               4d5h
kube-system               aws-node-927jv                           2/2     Running     0               4d5h
kube-system               aws-node-hhj66                           2/2     Running     0               4d5h
kube-system               aws-node-hpb57                           2/2     Running     0               4d5h
kube-system               aws-node-ng7nc                           2/2     Running     0               4d5h
kube-system               coredns-79b5589f65-fb64k                 1/1     Running     0               52d
kube-system               coredns-79b5589f65-fkkrw                 1/1     Running     0               52d
kube-system               ebs-csi-controller-77fcc9d855-55555      6/6     Running     9 (3d8h ago)    21d
kube-system               ebs-csi-controller-77fcc9d855-crrrr      6/6     Running     0               21d
kube-system               ebs-csi-node-d2qsw                       3/3     Running     0               21d
kube-system               ebs-csi-node-flw5z                       3/3     Running     0               21d
kube-system               ebs-csi-node-p5r5n                       3/3     Running     0               21d
kube-system               ebs-csi-node-tmdzd                       3/3     Running     0               21d
kube-system               ebs-csi-node-xpq88                       3/3     Running     0               21d
kube-system               efs-csi-controller-6f7f5f5787-ccccc      3/3     Running     0               52d
kube-system               efs-csi-controller-6f7f5f5787-xxxxx      3/3     Running     0               52d
kube-system               efs-csi-node-7zgx2                       3/3     Running     0               52d
kube-system               efs-csi-node-8hcmx                       3/3     Running     0               52d
kube-system               efs-csi-node-mhr9m                       3/3     Running     0               21d
kube-system               efs-csi-node-mzbfq                       3/3     Running     0               52d
kube-system               efs-csi-node-w859z                       3/3     Running     0               21d
kube-system               kube-proxy-28s6c                         1/1     Running     0               52d
kube-system               kube-proxy-4fjdc                         1/1     Running     0               21d
kube-system               kube-proxy-h47q6                         1/1     Running     0               52d
kube-system               kube-proxy-h5sjf                         1/1     Running     0               21d
kube-system               kube-proxy-pg5fx                         1/1     Running     0               52d
kube-system               metrics-server-5b787c6cf8-21356          1/1     Running     0               6h40m
dev-opensearch            opensearch-cluster-master-0              1/1     Running     0               52d
dev-opensearch            opensearch-cluster-master-1              1/1     Running     0               52d
dev-opensearch            opensearch-cluster-master-2              1/1     Running     0               52d
dev-rabbitmq              rabbitmq-0                               1/1     Running     0               9d
dev-rabbitmq              rabbitmq-1                               1/1     Running     0               9d
dev-rabbitmq              rabbitmq-2                               1/1     Running     0               9d
dev-redis                 redis-master-0                           1/1     Running     0               10d
dev-redis                 redis-replicas-0                         1/1     Running     0               3d8h
dev-redis                 redis-replicas-1                         1/1     Running     0               3d8h
dev-app                   app-app-enterprise-0                     2/2     Running     0               3d8h 
dev-app                   app-app-enterprise-1                     2/2     Running     0               3d8h
dev                       example-97f846598-hl6cc                  1/1     Running     0               28h
dev                       example-cron-28418300-d6xmw              0/1     Completed   0               12m
dev                       example-cron-28418305-c9hhj              0/1     Completed   0               7m4s
dev                       example-cron-28418310-chbfl              0/1     Completed   0               2m4s
dev                       example-post-upgrade-hlpmg               0/1     Completed   0               32h
dev                       ubuntu                                   1/1     Running     1 (3d11h ago)   10d
zero-trust                cloudflared-65bcdff7c6-gnp29             1/1     Running     0               2d13h
zero-trust                cloudflared-65bcdff7c6-grnmw             1/1     Running     0               2d13h
</code></pre>
<p>This is my hpa:</p>
<pre><code>kubectl get hpa --all-namespaces
NAMESPACE   NAME      REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
dev      example   Deployment/example   &lt;unknown&gt;/25%   1         11        1          24h
</code></pre>
<p>When I check my cluster logs:</p>
<pre><code>kubectl describe hpa --all-namespaces
Name:                                                  example
Namespace:                                             dev
Labels:                                                app.kubernetes.io/instance=example
                                                       app.kubernetes.io/managed-by=Helm
                                                       app.kubernetes.io/name=example
                                                       helm.sh/chart=example-23.0.1
Annotations:                                           meta.helm.sh/release-name: example
                                                       meta.helm.sh/release-namespace: dev
CreationTimestamp:                                     Thu, 11 Jan 2024 17:57:10 +0000
Reference:                                             Deployment/example
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  &lt;unknown&gt; / 25%
Min replicas:                                          1
Max replicas:                                          11
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                   From                       Message
  ----     ------                   ----                  ----                       -------
  Warning  FailedGetResourceMetric  19s (x5776 over 24h)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
</code></pre>
<p>metrics-server logs:</p>
<pre><code>kubectl logs deployment/metrics-server -n kube-system
I0112 15:51:32.519856       1 serving.go:342] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0112 15:51:32.996832       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0112 15:51:32.996894       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0112 15:51:32.996976       1 configmap_cafile_content.go:201] &quot;Starting controller&quot; name=&quot;client-ca::kube-system::extension-apiserver-authentication::client-ca-file&quot;
I0112 15:51:32.997058       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0112 15:51:32.997119       1 configmap_cafile_content.go:201] &quot;Starting controller&quot; name=&quot;client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file&quot;
I0112 15:51:32.997164       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0112 15:51:32.997877       1 dynamic_serving_content.go:131] &quot;Starting controller&quot; name=&quot;serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key&quot;
I0112 15:51:32.998109       1 secure_serving.go:267] Serving securely on [::]:8443
I0112 15:51:32.998136       1 tlsconfig.go:240] &quot;Starting DynamicServingCertificateController&quot;
W0112 15:51:33.000474       1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I0112 15:51:33.002798       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
I0112 15:51:33.097938       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0112 15:51:33.097937       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0112 15:51:33.097969       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0112 15:51:33.939463       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
I0112 15:51:37.866094       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
I0112 15:51:47.866103       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
I0112 15:51:57.866445       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
I0112 15:52:07.867043       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
I0112 15:52:17.865920       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
I0112 15:52:27.866276       1 server.go:187] &quot;Failed probe&quot; probe=&quot;metric-storage-ready&quot; err=&quot;no metrics to serve&quot;
</code></pre>
<p>Further queries yield no metrics</p>
<pre><code>kubectl top pod -A
error: Metrics API not available
kubectl top pods -n kube-system
error: Metrics API not available
</code></pre>
<p>Checking the metrics API, shows nothing:</p>
<pre><code>$kubectl api-versions | grep metrics
external.metrics.k8s.io/v1beta1
</code></pre>

CC BY-SA 4.0
2024-01-14T23:00:12.070Z
That means your metrics-server cannot reach kubelet for getting metrics. Make sure metrics-server can reach nodes on port 10250. You can also try to add `--kubelet-insecure-tls=true` to metrics-server args.
Vasilii Angapov
CC BY-SA 4.0
2024-01-15T14:58:37.367Z
Thank you @VasiliiAngapov for your suggestions, I can confirm this port is allowed. I had added the additional parameter of `--kubelet-insecure-tls=true`. Unfortunately, I am still seeing this error: `1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"`
Don Draper
CC BY-SA 4.0
2024-01-15T15:14:58.247Z
@VasiliiAngapov I should point out that the changes you suggested had a positive effect as I can now see the metric api. I've included that option at the very bottom of the post.
Don Draper
<kubernetes><terraform><helm><eks>
metrics-server on EKS - no metrics to server
Don Draper
# metrics-server on EKS - no metrics to server

(Asked by: username_0 on 2024-01-12)

I am attempting to setup metrics-server to server my 'horizon pod autoscaling' and manage my app pods scaling based on CPU. I am unable to get the metrics-server to run properly which fails with Failed probe" probe="metric-storage-ready" err="no metrics to serve. You can tell I am using EKS and managing the cluster using Terraform. Can anyone offer any insights?
Versions
  - EKS 1.27
  - Helm 3.12.0

This is my terraform script showing this specific piece of my cluster:
resource "kubernetes_namespace" "metric-server" {
  metadata {
    name = "${var.cluster_name}-metric-server"
  }
}

resource "helm_release" "metric_server" {
  name       = "metric-server"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "metric-server"
  namespace  = "${var.cluster_name}-metric-server"
  
  values = [
    "${file("./values/metrics-server.yaml")}"
  ]
}

The contents of ./values/metrics-server.yaml as follows.
spec:
  containers:
  - arg:
      - --metric-resolution=18s

This is my cluster (kube-system)
kubectl get po -n kube-system
NAME                                  READY   STATUS    RESTARTS       AGE
NAMESPACE                 NAME                                     READY   STATUS      RESTARTS        AGE
default                   reloader-reloader-7568f766fb-mkdzh       1/1     Running     0               3d8h
kube-system               aws-node-8w7lc                           2/2     Running     0               4d5h
kube-system               aws-node-927jv                           2/2     Running     0               4d5h
kube-system               aws-node-hhj66                           2/2     Running     0               4d5h
kube-system               aws-node-hpb57                           2/2     Running     0               4d5h
kube-system               aws-node-ng7nc                           2/2     Running     0               4d5h
kube-system               coredns-79b5589f65-fb64k                 1/1     Running     0               52d
kube-system               coredns-79b5589f65-fkkrw                 1/1     Running     0               52d
kube-system               ebs-csi-controller-77fcc9d855-55555      6/6     Running     9 (3d8h ago)    21d
kube-system               ebs-csi-controller-77fcc9d855-crrrr      6/6     Running     0               21d
kube-system               ebs-csi-node-d2qsw                       3/3     Running     0               21d
kube-system               ebs-csi-node-flw5z                       3/3     Running     0               21d
kube-system               ebs-csi-node-p5r5n                       3/3     Running     0               21d
kube-system               ebs-csi-node-tmdzd                       3/3     Running     0               21d
kube-system               ebs-csi-node-xpq88                       3/3     Running     0               21d
kube-system               efs-csi-controller-6f7f5f5787-ccccc      3/3     Running     0               52d
kube-system               efs-csi-controller-6f7f5f5787-xxxxx      3/3     Running     0               52d
kube-system               efs-csi-node-7zgx2                       3/3     Running     0               52d
kube-system               efs-csi-node-8hcmx                       3/3     Running     0               52d
kube-system               efs-csi-node-mhr9m                       3/3     Running     0               21d
kube-system               efs-csi-node-mzbfq                       3/3     Running     0               52d
kube-system               efs-csi-node-w859z                       3/3     Running     0               21d
kube-system               kube-proxy-28s6c                         1/1     Running     0               52d
kube-system               kube-proxy-4fjdc                         1/1     Running     0               21d
kube-system               kube-proxy-h47q6                         1/1     Running     0               52d
kube-system               kube-proxy-h5sjf                         1/1     Running     0               21d
kube-system               kube-proxy-pg5fx                         1/1     Running     0               52d
kube-system               metrics-server-5b787c6cf8-21356          1/1     Running     0               6h40m
dev-opensearch            opensearch-cluster-master-0              1/1     Running     0               52d
dev-opensearch            opensearch-cluster-master-1              1/1     Running     0               52d
dev-opensearch            opensearch-cluster-master-2              1/1     Running     0               52d
dev-rabbitmq              rabbitmq-0                               1/1     Running     0               9d
dev-rabbitmq              rabbitmq-1                               1/1     Running     0               9d
dev-rabbitmq              rabbitmq-2                               1/1     Running     0               9d
dev-redis                 redis-master-0                           1/1     Running     0               10d
dev-redis                 redis-replicas-0                         1/1     Running     0               3d8h
dev-redis                 redis-replicas-1                         1/1     Running     0               3d8h
dev-app                   app-app-enterprise-0                     2/2     Running     0               3d8h 
dev-app                   app-app-enterprise-1                     2/2     Running     0               3d8h
dev                       example-97f846598-hl6cc                  1/1     Running     0               28h
dev                       example-cron-28418300-d6xmw              0/1     Completed   0               12m
dev                       example-cron-28418305-c9hhj              0/1     Completed   0               7m4s
dev                       example-cron-28418310-chbfl              0/1     Completed   0               2m4s
dev                       example-post-upgrade-hlpmg               0/1     Completed   0               32h
dev                       ubuntu                                   1/1     Running     1 (3d11h ago)   10d
zero-trust                cloudflared-65bcdff7c6-gnp29             1/1     Running     0               2d13h
zero-trust                cloudflared-65bcdff7c6-grnmw             1/1     Running     0               2d13h

This is my hpa:
kubectl get hpa --all-namespaces
NAMESPACE   NAME      REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
dev      example   Deployment/example   /25%   1         11        1          24h

When I check my cluster logs:
kubectl describe hpa --all-namespaces
Name:                                                  example
Namespace:                                             dev
Labels:                                                app.kubernetes.io/instance=example
                                                       app.kubernetes.io/managed-by=Helm
                                                       app.kubernetes.io/name=example
                                                       helm.sh/chart=example-23.0.1
Annotations:                                           meta.helm.sh/release-name: example
                                                       meta.helm.sh/release-namespace: dev
CreationTimestamp:                                     Thu, 11 Jan 2024 17:57:10 +0000
Reference:                                             Deployment/example
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):   / 25%
Min replicas:                                          1
Max replicas:                                          11
Deployment pods:                                       1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetResourceMetric  the HPA was unable to compute the replica count: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
  Type     Reason                   Age                   From                       Message
  ----     ------                   ----                  ----                       -------
  Warning  FailedGetResourceMetric  19s (x5776 over 24h)  horizontal-pod-autoscaler  failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)

metrics-server logs:
kubectl logs deployment/metrics-server -n kube-system
I0112 15:51:32.519856       1 serving.go:342] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I0112 15:51:32.996832       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0112 15:51:32.996894       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0112 15:51:32.996976       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0112 15:51:32.997058       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0112 15:51:32.997119       1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0112 15:51:32.997164       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0112 15:51:32.997877       1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::apiserver.local.config/certificates/apiserver.crt::apiserver.local.config/certificates/apiserver.key"
I0112 15:51:32.998109       1 secure_serving.go:267] Serving securely on [::]:8443
I0112 15:51:32.998136       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
W0112 15:51:33.000474       1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I0112 15:51:33.002798       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:51:33.097938       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0112 15:51:33.097937       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0112 15:51:33.097969       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0112 15:51:33.939463       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:51:37.866094       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:51:47.866103       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:51:57.866445       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:52:07.867043       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:52:17.865920       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"
I0112 15:52:27.866276       1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"

Further queries yield no metrics
kubectl top pod -A
error: Metrics API not available
kubectl top pods -n kube-system
error: Metrics API not available

Checking the metrics API, shows nothing:
$kubectl api-versions | grep metrics
external.metrics.k8s.io/v1beta1

**Tags:** kubernetes, terraform, helm, eks

> **Comments:**
> **username_1**: That means your metrics-server cannot reach kubelet for getting metrics. Make sure metrics-server can reach nodes on port 10250. You can also try to add `--kubelet-insecure-tls=true` to metrics-server args.  
> **username_0**: Thank you @VasiliiAngapov for your suggestions, I can confirm this port is allowed. I had added the additional parameter of `--kubelet-insecure-tls=true`. Unfortunately, I am still seeing this error: `1 server.go:187] "Failed probe" probe="metric-storage-ready" err="no metrics to serve"`  
> **username_0**: @VasiliiAngapov I should point out that the changes you suggested had a positive effect as I can now see the metric api. I've included that option at the very bottom of the post.  

---

## 1 answers

### Answer by username_0 on 2024-01-17. Score: 0 | Accepted answer

I resolved this by using a different helm chart. I believe my original problem was caused by a version incompatibility.
resource "helm_release" "metrics_server" {
  name       = "metrics-server"
  repository = "https://kubernetes-sigs.github.io/metrics-server"
  chart      = "metrics-server"
  namespace  = "kube-system"
}


---

