<p>kubectl logs [pod-name] -c prometheus-server -n prometheus</p>
<p>level=error msg=&quot;Failed to apply configuration&quot; err=&quot;could not get SigV4 credentials: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for</p>
<p>check whether you missed the openID</p>

CC BY-SA 4.0
2024-03-14T06:25:32.837Z
2024-03-14T06:25:32.837Z
easy.yan
<p>I've had this issue in the past as well and I ended up noticing that my nodes were not joining the master.</p>
<p>Ensure that your nodes have joined the master and are in a ready state.</p>

CC BY-SA 4.0
2023-08-08T01:36:50.967Z
2023-08-08T04:53:53.653Z
2023-08-08T04:53:53.653Z
Freddy
<p>I followed the steps here <a href="https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html" rel="nofollow noreferrer">https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html</a></p>
<p>In a nutshell, I installed the Prometheus community helm chart on my EKS cluster in AWS <a href="https://github.com/prometheus-community/helm-charts" rel="nofollow noreferrer">https://github.com/prometheus-community/helm-charts</a>.</p>
<p>My Kubernetes version is 1.27</p>
<p>The cluster node group has two t3.large nodes (8gb memory, 2vCPU)</p>
<p>After creation, I have:</p>
<p>Pods:</p>
<pre><code>kubectl get pods -n prometheus

NAME                                                        READY   STATUS             RESTARTS         AGE
prometheus-alertmanager-0                            1/1     Running            0                142m
prometheus-kube-state-metrics-69b5df7dc4-kfqc2       1/1     Running            0                142m
prometheus-prometheus-node-exporter-dkksq            1/1     Running            0                142m
prometheus-prometheus-node-exporter-fzr5z            1/1     Running            0                142m
prometheus-prometheus-pushgateway-847c6f4d57-dz5wx   1/1     Running            0                142m
prometheus-server-5fbb47548d-sxm72                   1/2     CrashLoopBackOff   32 (3m58s ago)   142m
</code></pre>
<p>Pod logs:</p>
<pre><code> kubectl logs prometheus-server-5fbb47548d-sxm72 -n prometheus

....
level=error ts=2023-08-04T17:45:27.636671778Z caller=runutil.go:100 msg=&quot;function failed. Retrying in next tick&quot; err=&quot;trigger reload: reload request failed: Post \&quot;http://127.0.0.1:9090/-/reload\&quot;: dial tcp 127.0.0.1:9090: connect: connection refused&quot;
</code></pre>
<p>Deployments:</p>
<pre><code>kubectl get deployments -n prometheus
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-kube-state-metrics       1/1     1            1           144m
prometheus-prometheus-pushgateway   1/1     1            1           144m
prometheus-server                   0/1     1            0           144m
</code></pre>
<p>Describe failed deployment:</p>
<pre><code>kubectl describe deployments prometheus-server -n prometheus

...
 Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded
</code></pre>
<p>The EBS volumes have successfully bound.</p>
<p>It seems that the prometheus-server pod is stuck in CrashLoopBackOff because it can't connect to itself on 127.0.0.1:9090</p>
<p>I have been searching for an answer all day and trying different things but can't solve the issue.</p>
<p>My helm values looks like this:</p>
<pre><code>## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
##
serviceAccounts:
  server:
    name: amp-iamproxy-ingest-service-account
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam:###########:role/service-role/Amazon_EventBridge_Invoke_Api_Destination_###########
server:
  remoteWrite:
    - url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/#############/api/v1/remote_write
      sigv4:
        region: eu-west-1
      queue_config:
        max_samples_per_send: 1000
        max_shards: 200
        capacity: 2500
</code></pre>
<p>I can't figure this out,
please help! I'm sure someone else has struggled with the same issue because I can't find much info on this. Why is it failing? What Can I do to fix it? Thanks in advance</p>

<kubernetes><amazon-web-services><prometheus><kubernetes-helm>
Aws EKS Prometheus helm failed pod CrashLoopBackOff with log "trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\"
Harry
# Aws EKS Prometheus helm failed pod CrashLoopBackOff with log "trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\"

(Asked by: username_0 on 2023-08-04)

I followed the steps here https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-onboard-ingest-metrics-new-Prometheus.html
In a nutshell, I installed the Prometheus community helm chart on my EKS cluster in AWS https://github.com/prometheus-community/helm-charts.
My Kubernetes version is 1.27
The cluster node group has two t3.large nodes (8gb memory, 2vCPU)
After creation, I have:
Pods:
kubectl get pods -n prometheus

NAME                                                        READY   STATUS             RESTARTS         AGE
prometheus-alertmanager-0                            1/1     Running            0                142m
prometheus-kube-state-metrics-69b5df7dc4-kfqc2       1/1     Running            0                142m
prometheus-prometheus-node-exporter-dkksq            1/1     Running            0                142m
prometheus-prometheus-node-exporter-fzr5z            1/1     Running            0                142m
prometheus-prometheus-pushgateway-847c6f4d57-dz5wx   1/1     Running            0                142m
prometheus-server-5fbb47548d-sxm72                   1/2     CrashLoopBackOff   32 (3m58s ago)   142m

Pod logs:
 kubectl logs prometheus-server-5fbb47548d-sxm72 -n prometheus

....
level=error ts=2023-08-04T17:45:27.636671778Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:9090/-/reload\": dial tcp 127.0.0.1:9090: connect: connection refused"

Deployments:
kubectl get deployments -n prometheus
NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-kube-state-metrics       1/1     1            1           144m
prometheus-prometheus-pushgateway   1/1     1            1           144m
prometheus-server                   0/1     1            0           144m

Describe failed deployment:
kubectl describe deployments prometheus-server -n prometheus

...
 Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded

The EBS volumes have successfully bound.
It seems that the prometheus-server pod is stuck in CrashLoopBackOff because it can't connect to itself on 127.0.0.1:9090
I have been searching for an answer all day and trying different things but can't solve the issue.
My helm values looks like this:
## The following is a set of default values for prometheus server helm chart which enable remoteWrite to AMP
## For the rest of prometheus helm chart values see: https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus/values.yaml
##
serviceAccounts:
  server:
    name: amp-iamproxy-ingest-service-account
    annotations: 
      eks.amazonaws.com/role-arn: arn:aws:iam:###########:role/service-role/Amazon_EventBridge_Invoke_Api_Destination_###########
server:
  remoteWrite:
    - url: https://aps-workspaces.eu-west-1.amazonaws.com/workspaces/#############/api/v1/remote_write
      sigv4:
        region: eu-west-1
      queue_config:
        max_samples_per_send: 1000
        max_shards: 200
        capacity: 2500

I can't figure this out,
please help! I'm sure someone else has struggled with the same issue because I can't find much info on this. Why is it failing? What Can I do to fix it? Thanks in advance

**Tags:** kubernetes, amazon-web-services, prometheus, kubernetes-helm


---

## 2 answers

### Answer by username_1 on 2024-03-14. Score: 0

kubectl logs [pod-name] -c prometheus-server -n prometheus

level=error msg="Failed to apply configuration" err="could not get SigV4 credentials: WebIdentityErr: failed to retrieve credentials\ncaused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for

check whether you missed the openID


---

### Answer by username_2 on 2023-08-08. Score: 0

I've had this issue in the past as well and I ended up noticing that my nodes were not joining the master.

Ensure that your nodes have joined the master and are in a ready state.


---

