<p>A responder on the Grafana/Helm charts GitHub site (<a href="https://github.com/grafana/helm-charts/issues/2383" rel="nofollow noreferrer">https://github.com/grafana/helm-charts/issues/2383</a>) pointed out what was wrong with my config. Turns out I do not need the <code>root_url</code> for the grafana.ini. Once that was removed the issue has been resolved.</p>

CC BY-SA 4.0
2023-05-12T11:57:01.227Z
2023-05-12T11:57:01.227Z
Joseph Gagnon
<p>I have a Kubernetes cluster with the <code>kube-prometheus-stack</code> Helm chart (<a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="nofollow noreferrer">https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack</a>) installed. NGINX is used as the reverse proxy. We are using Kubernetes 1.24.</p>
<p>I am running into a problem when I attempt to access the Grafana dashboard. I am getting the &quot;too many redirects&quot; error. It can even be seen in the browser URL:</p>
<p><a href="https://i.sstatic.net/ALgjq.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ALgjq.png" alt="Too many redirects" /></a></p>
<p>I have tried to find what could be causing this and a solution to correct the problem. I found some info at the following site that seems to be very much related, but from what I can tell, what is discussed is specific to a more direct installation, rather than within the context of Kubernetes, Helm charts and container abstractions.</p>
<p><a href="https://github.com/grafana/grafana/blob/main/docs/sources/tutorials/run-grafana-behind-a-proxy/index.md" rel="nofollow noreferrer">https://github.com/grafana/grafana/blob/main/docs/sources/tutorials/run-grafana-behind-a-proxy/index.md</a></p>
<p>and</p>
<p><a href="https://grafana.com/tutorials/run-grafana-behind-a-proxy/" rel="nofollow noreferrer">https://grafana.com/tutorials/run-grafana-behind-a-proxy/</a></p>
<p>I think I can see a parallel between some of what's mentioned in the above articles (they have a lot of the same information) and my Helm configuration (<code>values.yaml</code>). However, I have no idea what is being referred to regarding NGINX configuration. I'm not sure where to look to see the NGINX config or how to modify it. There is mention of setting up an NGINX rewrite directive, but I have no idea what that is, how it can be done, or if I even need it.</p>
<p>Here is my Helm chart <code>values.yaml</code> if it's of any help.</p>
<pre><code>kube-prometheus-stack:
  defaultRules:
    create: true
    rules:
      alertmanager: true
      etcd: true
      configReloaders: true
      general: true
      k8s: true
      kubeApiserverAvailability: true
      kubeApiserverBurnrate: true
      kubeApiserverHistogram: true
      kubeApiserverSlos: true
      kubeControllerManager: true
      kubelet: true
      kubeProxy: true
      kubePrometheusGeneral: true
      kubePrometheusNodeRecording: true
      kubernetesApps: true
      kubernetesResources: true
      kubernetesStorage: true
      kubernetesSystem: true
      kubeSchedulerAlerting: true
      kubeSchedulerRecording: true
      kubeStateMetrics: true
      network: true
      node: true
      nodeExporterAlerting: true
      nodeExporterRecording: true
      prometheus: true
      prometheusOperator: true

  prometheus:
    enabled: true
    ingress:
      enabled: true
      ingressClassName: nginx
      hosts:
        - prometheus.nnpi.beast-code.com
      paths:
        - /
      pathType: ImplementationSpecific

  grafana:
    enabled: true
    ingress:
      enabled: true
      ingressClassName: nginx
      hosts:
        - grafana.nnpi.beast-code.com
      path: /
    persistence:
      enabled: true
      # size: 10Gi
    grafana.ini:
      server:
        root_url: grafana.nnpi.beast-code.com

  alertmanager:
    enabled: true
    ingress:
      enabled: true
      ingressClassName: nginx
      hosts:
        - alerts.nnpi.beast-code.com
      paths:
        - /
      pathType: ImplementationSpecific
    config:
      global:
        slack_api_url: &lt;path:arc/data/alertmanager#SLACK_API_URL&gt;
      route:
        receiver: &quot;slack-default&quot;
        group_by:
          - alertname
          - cluster
          - service
        group_wait: 30s # 30s
        group_interval: 5m # 5m
        repeat_interval: 1h # 4h
        routes:
          - receiver: &quot;null&quot;
            matchers:
              - alertname =~ &quot;InfoInhibitor|Watchdog&quot;
          - receiver: &quot;slack-warn-critical&quot;
            matchers:
              - severity =~ &quot;warning|critical&quot;
            continue: true
      receivers:
        - name: &quot;null&quot;
        - name: &quot;slack-default&quot;
          slack_configs:
            - send_resolved: true # false
              channel: &quot;#notify-arc-alerts&quot;
              text: |-
                {{range .Alerts}}
                Alert: {{.Labels.alertname}}, Severity: {{.Labels.severity}}
                Summary: {{.Annotations.summary}}
                Description: {{.Annotations.description}}
                {{end}}
        - name: &quot;slack-warn-critical&quot;
          slack_configs:
            - send_resolved: true # false
              channel: &quot;#notify-arc-alerts&quot;
              text: |-
                {{range .Alerts}}
                Alert: {{.Labels.alertname}}, Severity: {{.Labels.severity}}
                Summary: {{.Annotations.summary}}
                Description: {{.Annotations.description}}
                {{end}}
</code></pre>
<p>Here is the generated ingress:</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-grafana
  namespace: prometheus
  uid: df852ca5-897e-4d5c-bd6b-09b4a4a45910
  resourceVersion: '111566747'
  generation: 1
  creationTimestamp: '2022-11-09T19:27:17Z'
  labels:
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 9.3.1
    helm.sh/chart: grafana-6.48.2
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: &gt;
      {&quot;apiVersion&quot;:&quot;networking.k8s.io/v1&quot;,&quot;kind&quot;:&quot;Ingress&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;labels&quot;:{&quot;app.kubernetes.io/instance&quot;:&quot;prometheus&quot;,&quot;app.kubernetes.io/managed-by&quot;:&quot;Helm&quot;,&quot;app.kubernetes.io/name&quot;:&quot;grafana&quot;,&quot;app.kubernetes.io/version&quot;:&quot;9.3.1&quot;,&quot;helm.sh/chart&quot;:&quot;grafana-6.48.2&quot;},&quot;name&quot;:&quot;prometheus-grafana&quot;,&quot;namespace&quot;:&quot;prometheus&quot;},&quot;spec&quot;:{&quot;ingressClassName&quot;:&quot;nginx&quot;,&quot;rules&quot;:[{&quot;host&quot;:&quot;grafana.nnpi.beast-code.com&quot;,&quot;http&quot;:{&quot;paths&quot;:[{&quot;backend&quot;:{&quot;service&quot;:{&quot;name&quot;:&quot;prometheus-grafana&quot;,&quot;port&quot;:{&quot;number&quot;:80}}},&quot;path&quot;:&quot;/&quot;,&quot;pathType&quot;:&quot;Prefix&quot;}]}}]}}
  managedFields:
    - manager: argocd-application-controller
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2022-11-09T19:27:17Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/version: {}
            f:helm.sh/chart: {}
        f:spec:
          f:ingressClassName: {}
          f:rules: {}
    - manager: nginx-ingress-controller
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2022-11-09T19:27:56Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:loadBalancer:
            f:ingress: {}
      subresource: status
  selfLink: &gt;-
    /apis/networking.k8s.io/v1/namespaces/prometheus/ingresses/prometheus-grafana
status:
  loadBalancer:
    ingress:
      - ip: 10.107.160.141
spec:
  ingressClassName: nginx
  rules:
    - host: grafana.nnpi.beast-code.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: prometheus-grafana
                port:
                  number: 80
</code></pre>
<p>I have also included a section of the <code>nginx.conf</code> file from one of the nginx controller pods running in the cluster. I do not understand this at all, but I'm guessing that a combination of ArgoCD, Helm and Kubernetes is taking care of setting this up.</p>
<pre><code>## start server grafana.nnpi.beast-code.com
server {
        server_name grafana.nnpi.beast-code.com ;

        listen 80  ;
        listen [::]:80  ;
        listen 443  ssl http2 ;
        listen [::]:443  ssl http2 ;

        set $proxy_upstream_name &quot;-&quot;;

        ssl_certificate_by_lua_block {
                certificate.call()
        }

        location / {

                set $namespace      &quot;prometheus&quot;;
                set $ingress_name   &quot;prometheus-grafana&quot;;
                set $service_name   &quot;prometheus-grafana&quot;;
                set $service_port   &quot;80&quot;;
                set $location_path  &quot;/&quot;;
                set $global_rate_limit_exceeding n;

                rewrite_by_lua_block {
                        lua_ingress.rewrite({
                                force_ssl_redirect = true,
                                ssl_redirect = true,
                                force_no_ssl_redirect = false,
                                preserve_trailing_slash = false,
                                use_port_in_redirects = false,
                                global_throttle = { namespace = &quot;&quot;, limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
                        })
                        balancer.rewrite()
                        plugins.run()
                }

                # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
                # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
                # other authentication method such as basic auth or external auth useless - all requests will be allowed.
                #access_by_lua_block {
                #}

                header_filter_by_lua_block {
                        lua_ingress.header()
                        plugins.run()
                }

                body_filter_by_lua_block {
                        plugins.run()
                }

                log_by_lua_block {
                        balancer.log()

                        monitor.call()

                        plugins.run()
                }

                port_in_redirect off;

                set $balancer_ewma_score -1;
                set $proxy_upstream_name &quot;prometheus-prometheus-grafana-80&quot;;
                set $proxy_host          $proxy_upstream_name;
                set $pass_access_scheme  $scheme;

                set $pass_server_port    $server_port;

                set $best_http_host      $http_host;
                set $pass_port           $pass_server_port;

                set $proxy_alternative_upstream_name &quot;&quot;;

                client_max_body_size                    0;

                proxy_set_header Host                   $best_http_host;

                # Pass the extracted client certificate to the backend

                # Allow websocket connections
                proxy_set_header                        Upgrade           $http_upgrade;

                proxy_set_header                        Connection        $connection_upgrade;

                proxy_set_header X-Request-ID           $req_id;
                proxy_set_header X-Real-IP              $remote_addr;

                proxy_set_header X-Forwarded-For        $remote_addr;

                proxy_set_header X-Forwarded-Host       $best_http_host;
                proxy_set_header X-Forwarded-Port       $pass_port;
                proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
                proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

                proxy_set_header X-Scheme               $pass_access_scheme;

                # Pass the original X-Forwarded-For
                proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

                # mitigate HTTPoxy Vulnerability
                # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                proxy_set_header Proxy                  &quot;&quot;;

                # Custom headers to proxied server

                proxy_connect_timeout                   5s;
                proxy_send_timeout                      60s;
                proxy_read_timeout                      60s;

                proxy_buffering                         off;
                proxy_buffer_size                       64k;
                proxy_buffers                           4 64k;

                proxy_max_temp_file_size                1024m;

                proxy_request_buffering                 on;
                proxy_http_version                      1.1;

                proxy_cookie_domain                     off;
                proxy_cookie_path                       off;

                # In case of errors try the next upstream server before returning an error
                proxy_next_upstream                     error timeout;
                proxy_next_upstream_timeout             0;
                proxy_next_upstream_tries               3;

                proxy_pass http://upstream_balancer;

                proxy_redirect                          off;

        }

}
## end server grafana.nnpi.beast-code.com
</code></pre>
<p>I would assume (and hope) that it would not be necessary to somehow modify this directly and that instead, modifying some upstream config (probably in the Helm chart <code>values.yaml</code>) would be propagated downward.</p>

<kubernetes><helm><nginx><grafana>
kube-prometheus-stack/grafana - Getting "too many redirects" error
Joseph Gagnon
# kube-prometheus-stack/grafana - Getting "too many redirects" error

(Asked by: username_0 on 2023-04-20)

I have a Kubernetes cluster with the kube-prometheus-stack Helm chart (https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) installed. NGINX is used as the reverse proxy. We are using Kubernetes 1.24.
I am running into a problem when I attempt to access the Grafana dashboard. I am getting the "too many redirects" error. It can even be seen in the browser URL:

I have tried to find what could be causing this and a solution to correct the problem. I found some info at the following site that seems to be very much related, but from what I can tell, what is discussed is specific to a more direct installation, rather than within the context of Kubernetes, Helm charts and container abstractions.
https://github.com/grafana/grafana/blob/main/docs/sources/tutorials/run-grafana-behind-a-proxy/index.md
and
https://grafana.com/tutorials/run-grafana-behind-a-proxy/
I think I can see a parallel between some of what's mentioned in the above articles (they have a lot of the same information) and my Helm configuration (values.yaml). However, I have no idea what is being referred to regarding NGINX configuration. I'm not sure where to look to see the NGINX config or how to modify it. There is mention of setting up an NGINX rewrite directive, but I have no idea what that is, how it can be done, or if I even need it.
Here is my Helm chart values.yaml if it's of any help.
kube-prometheus-stack:
  defaultRules:
    create: true
    rules:
      alertmanager: true
      etcd: true
      configReloaders: true
      general: true
      k8s: true
      kubeApiserverAvailability: true
      kubeApiserverBurnrate: true
      kubeApiserverHistogram: true
      kubeApiserverSlos: true
      kubeControllerManager: true
      kubelet: true
      kubeProxy: true
      kubePrometheusGeneral: true
      kubePrometheusNodeRecording: true
      kubernetesApps: true
      kubernetesResources: true
      kubernetesStorage: true
      kubernetesSystem: true
      kubeSchedulerAlerting: true
      kubeSchedulerRecording: true
      kubeStateMetrics: true
      network: true
      node: true
      nodeExporterAlerting: true
      nodeExporterRecording: true
      prometheus: true
      prometheusOperator: true

  prometheus:
    enabled: true
    ingress:
      enabled: true
      ingressClassName: nginx
      hosts:
        - prometheus.nnpi.beast-code.com
      paths:
        - /
      pathType: ImplementationSpecific

  grafana:
    enabled: true
    ingress:
      enabled: true
      ingressClassName: nginx
      hosts:
        - grafana.nnpi.beast-code.com
      path: /
    persistence:
      enabled: true
      # size: 10Gi
    grafana.ini:
      server:
        root_url: grafana.nnpi.beast-code.com

  alertmanager:
    enabled: true
    ingress:
      enabled: true
      ingressClassName: nginx
      hosts:
        - alerts.nnpi.beast-code.com
      paths:
        - /
      pathType: ImplementationSpecific
    config:
      global:
        slack_api_url: 
      route:
        receiver: "slack-default"
        group_by:
          - alertname
          - cluster
          - service
        group_wait: 30s # 30s
        group_interval: 5m # 5m
        repeat_interval: 1h # 4h
        routes:
          - receiver: "null"
            matchers:
              - alertname =~ "InfoInhibitor|Watchdog"
          - receiver: "slack-warn-critical"
            matchers:
              - severity =~ "warning|critical"
            continue: true
      receivers:
        - name: "null"
        - name: "slack-default"
          slack_configs:
            - send_resolved: true # false
              channel: "#notify-arc-alerts"
              text: |-
                {{range .Alerts}}
                Alert: {{.Labels.alertname}}, Severity: {{.Labels.severity}}
                Summary: {{.Annotations.summary}}
                Description: {{.Annotations.description}}
                {{end}}
        - name: "slack-warn-critical"
          slack_configs:
            - send_resolved: true # false
              channel: "#notify-arc-alerts"
              text: |-
                {{range .Alerts}}
                Alert: {{.Labels.alertname}}, Severity: {{.Labels.severity}}
                Summary: {{.Annotations.summary}}
                Description: {{.Annotations.description}}
                {{end}}

Here is the generated ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-grafana
  namespace: prometheus
  uid: df852ca5-897e-4d5c-bd6b-09b4a4a45910
  resourceVersion: '111566747'
  generation: 1
  creationTimestamp: '2022-11-09T19:27:17Z'
  labels:
    app.kubernetes.io/instance: prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 9.3.1
    helm.sh/chart: grafana-6.48.2
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{},"labels":{"app.kubernetes.io/instance":"prometheus","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"grafana","app.kubernetes.io/version":"9.3.1","helm.sh/chart":"grafana-6.48.2"},"name":"prometheus-grafana","namespace":"prometheus"},"spec":{"ingressClassName":"nginx","rules":[{"host":"grafana.nnpi.beast-code.com","http":{"paths":[{"backend":{"service":{"name":"prometheus-grafana","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}
  managedFields:
    - manager: argocd-application-controller
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2022-11-09T19:27:17Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:labels:
            .: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/version: {}
            f:helm.sh/chart: {}
        f:spec:
          f:ingressClassName: {}
          f:rules: {}
    - manager: nginx-ingress-controller
      operation: Update
      apiVersion: networking.k8s.io/v1
      time: '2022-11-09T19:27:56Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:status:
          f:loadBalancer:
            f:ingress: {}
      subresource: status
  selfLink: >-
    /apis/networking.k8s.io/v1/namespaces/prometheus/ingresses/prometheus-grafana
status:
  loadBalancer:
    ingress:
      - ip: 10.107.160.141
spec:
  ingressClassName: nginx
  rules:
    - host: grafana.nnpi.beast-code.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: prometheus-grafana
                port:
                  number: 80

I have also included a section of the nginx.conf file from one of the nginx controller pods running in the cluster. I do not understand this at all, but I'm guessing that a combination of ArgoCD, Helm and Kubernetes is taking care of setting this up.
## start server grafana.nnpi.beast-code.com
server {
        server_name grafana.nnpi.beast-code.com ;

        listen 80  ;
        listen [::]:80  ;
        listen 443  ssl http2 ;
        listen [::]:443  ssl http2 ;

        set $proxy_upstream_name "-";

        ssl_certificate_by_lua_block {
                certificate.call()
        }

        location / {

                set $namespace      "prometheus";
                set $ingress_name   "prometheus-grafana";
                set $service_name   "prometheus-grafana";
                set $service_port   "80";
                set $location_path  "/";
                set $global_rate_limit_exceeding n;

                rewrite_by_lua_block {
                        lua_ingress.rewrite({
                                force_ssl_redirect = true,
                                ssl_redirect = true,
                                force_no_ssl_redirect = false,
                                preserve_trailing_slash = false,
                                use_port_in_redirects = false,
                                global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
                        })
                        balancer.rewrite()
                        plugins.run()
                }

                # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
                # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
                # other authentication method such as basic auth or external auth useless - all requests will be allowed.
                #access_by_lua_block {
                #}

                header_filter_by_lua_block {
                        lua_ingress.header()
                        plugins.run()
                }

                body_filter_by_lua_block {
                        plugins.run()
                }

                log_by_lua_block {
                        balancer.log()

                        monitor.call()

                        plugins.run()
                }

                port_in_redirect off;

                set $balancer_ewma_score -1;
                set $proxy_upstream_name "prometheus-prometheus-grafana-80";
                set $proxy_host          $proxy_upstream_name;
                set $pass_access_scheme  $scheme;

                set $pass_server_port    $server_port;

                set $best_http_host      $http_host;
                set $pass_port           $pass_server_port;

                set $proxy_alternative_upstream_name "";

                client_max_body_size                    0;

                proxy_set_header Host                   $best_http_host;

                # Pass the extracted client certificate to the backend

                # Allow websocket connections
                proxy_set_header                        Upgrade           $http_upgrade;

                proxy_set_header                        Connection        $connection_upgrade;

                proxy_set_header X-Request-ID           $req_id;
                proxy_set_header X-Real-IP              $remote_addr;

                proxy_set_header X-Forwarded-For        $remote_addr;

                proxy_set_header X-Forwarded-Host       $best_http_host;
                proxy_set_header X-Forwarded-Port       $pass_port;
                proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
                proxy_set_header X-Forwarded-Scheme     $pass_access_scheme;

                proxy_set_header X-Scheme               $pass_access_scheme;

                # Pass the original X-Forwarded-For
                proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;

                # mitigate HTTPoxy Vulnerability
                # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                proxy_set_header Proxy                  "";

                # Custom headers to proxied server

                proxy_connect_timeout                   5s;
                proxy_send_timeout                      60s;
                proxy_read_timeout                      60s;

                proxy_buffering                         off;
                proxy_buffer_size                       64k;
                proxy_buffers                           4 64k;

                proxy_max_temp_file_size                1024m;

                proxy_request_buffering                 on;
                proxy_http_version                      1.1;

                proxy_cookie_domain                     off;
                proxy_cookie_path                       off;

                # In case of errors try the next upstream server before returning an error
                proxy_next_upstream                     error timeout;
                proxy_next_upstream_timeout             0;
                proxy_next_upstream_tries               3;

                proxy_pass http://upstream_balancer;

                proxy_redirect                          off;

        }

}
## end server grafana.nnpi.beast-code.com

I would assume (and hope) that it would not be necessary to somehow modify this directly and that instead, modifying some upstream config (probably in the Helm chart values.yaml) would be propagated downward.

**Tags:** kubernetes, helm, nginx, grafana


---

## 1 answers

### Answer by username_0 on 2023-05-12. Score: 0 | Accepted answer

A responder on the Grafana/Helm charts GitHub site (<https://github.com/grafana/helm-charts/issues/2383>) pointed out what was wrong with my config. Turns out I do not need the `root_url` for the grafana.ini. Once that was removed the issue has been resolved.


---

