<p>You need to send bearer token as well, like this:</p>
<pre><code>- job_name: 'kubelet'
        kubernetes_sd_configs:
        - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true 
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
</code></pre>

CC BY-SA 4.0
2023-03-13T01:16:11.340Z
Thanks Shanmuga! That changed the response from "server returned HTTP status 401 Unauthorized" to "server returned HTTP status 403 Forbidden". 

However, I still don't know why it is not authorized
codedread
CC BY-SA 4.0
2023-03-05T18:32:21.733Z
2023-03-05T18:32:48.880Z
2023-03-05T18:32:48.880Z
Shanmuga Raj
<p>I have a kubernetes (not minikube) cluster (1.25) installed on a dedicated machine (1-node cluster). I installed Prometheus. I am using a ServiceAccount, Role, ClusterRoleBinding like this:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Namespace
metadata:
    name: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [&quot;&quot;]
  resources:
  - nodes
  - services
  - endpoints
  - pods
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]
- nonResourceURLs:
  - &quot;/metrics&quot;
  verbs: [&quot;get&quot;]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring
</code></pre>
<p>with my Deployment resource containing <code>serviceAccountName: prometheus</code>.</p>
<p>I am getting some metrics (from Prometheus and some Istio scrape configs), but am not able to get the k8s node metrics (like CPU, etc). I tried this:</p>
<pre class="lang-yaml prettyprint-override"><code>      - job_name: 'kubelet'
        kubernetes_sd_configs:
        - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
</code></pre>
<p>I get an error in my Prometheus targets dashboard (<a href="http://192.168.0.121:30000/targets?search=" rel="nofollow noreferrer">http://192.168.0.121:30000/targets?search=</a> of:</p>
<p><code>Get &quot;https://192.168.0.121:10250/metrics&quot;: x509: cannot validate certificate for 192.168.0.121 because it doesn't contain any IP SANs</code></p>
<p>Adding <code>insecure_skip_verify: true</code> turns this error into:</p>
<p><code>server returned HTTP status 401 Unauthorized</code></p>
<p>My KubeletConfiguration:</p>
<pre class="lang-bash prettyprint-override"><code>% kubectl describe configmap kubelet-config -n kube-system
Name:         kubelet-config
Namespace:    kube-system
Labels:       &lt;none&gt;
Annotations:  kubeadm.kubernetes.io/component-config.hash: ...

Data
====
kubelet:
----
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: &quot;0&quot;
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s
</code></pre>
<p>My kubeadm-config is:</p>
<pre class="lang-bash prettyprint-override"><code>% kubectl describe configmap kubeadm-config -n kube-system
Name:         kubeadm-config
Namespace:    kube-system
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;

Data
====
ClusterConfiguration:
----
apiServer:
  certSANs:
  - &lt;my tailscale ip address&gt;
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.0.121:6443
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.25.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}
</code></pre>
<p>which was created by</p>
<p><code>kubeadm init --control-plane-endpoint=192.168.0.121 --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=&lt;my tailnet ip address&gt;</code></p>
<p>Any insight would be greatly appreciated!</p>

<kubernetes><prometheus><kubelet>
Prometheus scrape config for kubelet node role not auth'ing
codedread
# Prometheus scrape config for kubelet node role not auth'ing

(Asked by: username_0 on 2023-01-29)

I have a kubernetes (not minikube) cluster (1.25) installed on a dedicated machine (1-node cluster). I installed Prometheus. I am using a ServiceAccount, Role, ClusterRoleBinding like this:
apiVersion: v1
kind: Namespace
metadata:
    name: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs:
  - "/metrics"
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

with my Deployment resource containing serviceAccountName: prometheus.
I am getting some metrics (from Prometheus and some Istio scrape configs), but am not able to get the k8s node metrics (like CPU, etc). I tried this:
      - job_name: 'kubelet'
        kubernetes_sd_configs:
        - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

I get an error in my Prometheus targets dashboard (http://192.168.0.121:30000/targets?search= of:
Get "https://192.168.0.121:10250/metrics": x509: cannot validate certificate for 192.168.0.121 because it doesn't contain any IP SANs
Adding insecure_skip_verify: true turns this error into:
server returned HTTP status 401 Unauthorized
My KubeletConfiguration:
% kubectl describe configmap kubelet-config -n kube-system
Name:         kubelet-config
Namespace:    kube-system
Labels:       
Annotations:  kubeadm.kubernetes.io/component-config.hash: ...

Data
====
kubelet:
----
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

My kubeadm-config is:
% kubectl describe configmap kubeadm-config -n kube-system
Name:         kubeadm-config
Namespace:    kube-system
Labels:       
Annotations:  

Data
====
ClusterConfiguration:
----
apiServer:
  certSANs:
  - 
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.0.121:6443
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.25.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

which was created by
kubeadm init --control-plane-endpoint=192.168.0.121 --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=
Any insight would be greatly appreciated!

**Tags:** kubernetes, prometheus, kubelet


---

## 1 answers

### Answer by username_1 on 2023-03-05. Score: 1

You need to send bearer token as well, like this:
- job_name: 'kubelet'
        kubernetes_sd_configs:
        - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true 
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

> **Comments:**
> **username_0**: Thanks Shanmuga! That changed the response from "server returned HTTP status 401 Unauthorized" to "server returned HTTP status 403 Forbidden".   However, I still don't know why it is not authorized  

---

