<p>I managed to solve the issue by changing the configuration of <code>nodelocaldns</code>:</p>
<pre><code>kubectl edit configmap -n kube-system nodelocaldns
</code></pre>
<p>change from</p>
<pre><code>cluster.local:53 {
    errors
    cache {
        success 9984 30
        denial 9984 5
    }
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
    health 169.254.25.10:9254
}
in-addr.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
ip6.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
.:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . /etc/resolv.conf
    prometheus :9253
}
</code></pre>
<p>to this:</p>
<pre><code>cluster.local:53 {
    errors
    cache {
        success 9984 30
        denial 9984 5
    }
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
    health 169.254.25.10:9254
}
in-addr.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
ip6.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
.:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
</code></pre>
<p><em>look at the last block</em></p>
<p>I am curious what caused this issue in the first place?</p>

CC BY-SA 4.0
2024-08-19T11:05:37.420Z
2024-08-19T11:05:37.420Z
badger
<p>In Kubernetes there is a convention that the cluster DNS service is on the <code>.10</code> IP address.</p>
<p>So it's expected that you'll configure the ClusterIP for CoreDNS to request that particular IP.</p>
<p>If not, you'll need to configure kubelet on each node to change what it injects into the <code>/etc/resolv.conf</code> file of most pods.</p>

CC BY-SA 4.0
2024-08-19T20:42:31.023Z
Slightly more detail: https://serverfault.com/a/1158944/431900
benjimin
CC BY-SA 4.0
2024-08-19T20:23:02.783Z
2024-08-19T20:28:53.897Z
2024-08-19T20:28:53.897Z
benjimin
<p>I setup a Kubernetes cluster using Kubespray. it works flawlessly until recently I encountered something strange.</p>
<p>I created a BIND DNS deployment using the following manifest.</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: bind
  labels:
    app: bind
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bind
  template:
    metadata:
      labels:
        app: bind
    spec:
      containers:
      - name: bind
        image: sameersbn/bind:9.16.1-20200524
        ports:
        - containerPort: 53
          name: tcp
          protocol: TCP 
        - containerPort: 53
          name: udp
          protocol: UDP
        - containerPort: 10000
          name: web
          protocol: TCP
        volumeMounts:
        - name: bind-data
          mountPath: /data
      volumes:
      - name: bind-data
        hostPath:
          path: /data
---
apiVersion: v1
kind: Service
metadata:
  name: bind
spec:
  type: ClusterIP
  ports:
  - port: 53
    name: tcp
    targetPort: tcp
    protocol: TCP
  - port: 53
    name: udp
    targetPort: udp
    protocol: UDP
  - port: 10000
    name: web
    targetPort: web
    protocol: TCP
  selector:
    app: bind
</code></pre>
<p>Then I've defined a master zone and created a forward dns record in it.</p>
<pre><code>test.arman.local -&gt; 192.168.230.216 
</code></pre>
<p>Next I have configured the configmap of coredns to use it:</p>
<pre><code>kubectl edit configmap -n kube-system coredns

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    arman.local:53 {
        errors
        cache 30
        forward . 10.233.28.178
    }
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;data&quot;:{&quot;Corefile&quot;:&quot;.:53 {\n    errors {\n    }\n    health {\n        lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n      prefer_udp\n      max_concurrent 1000\n    }\n    cache 30\n\n    loop\n    reload\n    loadbalance\n}\n&quot;},&quot;kind&quot;:&quot;ConfigMap&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;labels&quot;:{&quot;addonmanager.kubernetes.io/mode&quot;:&quot;EnsureExists&quot;},&quot;name&quot;:&quot;coredns&quot;,&quot;namespace&quot;:&quot;kube-system&quot;}}
  creationTimestamp: &quot;2024-08-08T10:26:32Z&quot;
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: coredns
  namespace: kube-system
  resourceVersion: &quot;2135167&quot;
  uid: 5457c863-3a34-4490-badb-036f4f0e1739
</code></pre>
<p>The service of BIND deployment is like:</p>
<pre><code>kubectl get svc                 
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
bind                          ClusterIP   10.233.28.178   &lt;none&gt;        53/TCP,53/UDP,10000/TCP      44h
</code></pre>
<p><em>The problem is it can't resolve <code>test.arman.local</code></em></p>
<p>I've started to debug the problem I launched a <code>busybox</code> and used <code>nslookup</code></p>
<pre><code>kubectl run -it busybox --image=busybox:1.28 --restart=Never -- sh
</code></pre>
<p>Here is the content of <code>/etc/resolv.conf</code> of busybox</p>
<pre><code>/ # cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.25.10
options ndots:5
</code></pre>
<p><strong>I don't know what IP <code>169.254.25.10</code> is for because the service of dnsCore has different IP</strong>:</p>
<pre><code>kubectl get svc -n kube-system 
NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
coredns   ClusterIP   10.233.0.3   &lt;none&gt;        53/UDP,53/TCP,9153/TCP   10d
</code></pre>
<p>In the busybox I can successfully resolve the <code>test.arman.local</code> using the following command:</p>
<pre><code>nslookup test.arman.local 10.233.28.178
Server:    10.233.28.178
Address 1: 10.233.28.178

Name:      test.arman.local
Address 1: 192.168.230.216
/ # ^C
/ # cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.25.10
options ndots:5
</code></pre>
<p>Or even use this:</p>
<pre><code>nslookup test.arman.local 10.233.0.3 # This is for coredns
Server:    10.233.0.3
Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local

Name:      test.arman.local
Address 1: 192.168.230.216
</code></pre>
<p>We are using DNS/calico, for the sake of completeness I post the pods in kube-system:</p>
<pre><code> kubectl get pods -n kube-system 
NAME                                      READY   STATUS    RESTARTS        AGE
calico-kube-controllers-c7cc688f8-rb59q   1/1     Running   1 (7d21h ago)   10d
calico-node-5dglq                         1/1     Running   0               10d
calico-node-c592m                         1/1     Running   0               10d
calico-node-hbfqj                         1/1     Running   0               10d
calico-node-s6jrl                         1/1     Running   0               10d
coredns-857f949bf7-bprlf                  1/1     Running   0               65m
coredns-857f949bf7-r9p87                  1/1     Running   0               65m
dns-autoscaler-6ffb84bd6-fgdtz            1/1     Running   0               10d
kube-apiserver-node1                      1/1     Running   1               10d
kube-apiserver-node2                      1/1     Running   1               10d
kube-controller-manager-node1             1/1     Running   4 (7d21h ago)   10d
kube-controller-manager-node2             1/1     Running   7 (6d19h ago)   10d
kube-proxy-hb2wn                          1/1     Running   0               10d
kube-proxy-j5pfg                          1/1     Running   0               10d
kube-proxy-qjj4n                          1/1     Running   0               10d
kube-proxy-rp9mt                          1/1     Running   0               10d
kube-scheduler-node1                      1/1     Running   4 (6d19h ago)   10d
kube-scheduler-node2                      1/1     Running   6 (6d19h ago)   10d
nginx-proxy-node3                         1/1     Running   0               10d
nginx-proxy-node4                         1/1     Running   0               10d
nodelocaldns-6sztv                        1/1     Running   0               10d
nodelocaldns-bpt7s                        1/1     Running   0               10d
nodelocaldns-n4rsk                        1/1     Running   0               10d
nodelocaldns-n8q9w                        1/1     Running   0               10d
</code></pre>
<p>UPDATE: I found out it has something to do with <code>nodelocaldns</code> but I can not resolve the issue</p>

CC BY-SA 4.0
2024-08-19T20:25:03.547Z
Probable duplicate of e.g. https://serverfault.com/questions/1009907/kubernetes-pod-etc-resolv-conf-has-the-wrong-nameserver
benjimin
<kubernetes>
CoreDNS is not able to resolve names
badger
# CoreDNS is not able to resolve names

(Asked by: username_0 on 2024-08-19)

I setup a Kubernetes cluster using Kubespray. it works flawlessly until recently I encountered something strange.
I created a BIND DNS deployment using the following manifest.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bind
  labels:
    app: bind
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bind
  template:
    metadata:
      labels:
        app: bind
    spec:
      containers:
      - name: bind
        image: sameersbn/bind:9.16.1-20200524
        ports:
        - containerPort: 53
          name: tcp
          protocol: TCP 
        - containerPort: 53
          name: udp
          protocol: UDP
        - containerPort: 10000
          name: web
          protocol: TCP
        volumeMounts:
        - name: bind-data
          mountPath: /data
      volumes:
      - name: bind-data
        hostPath:
          path: /data
---
apiVersion: v1
kind: Service
metadata:
  name: bind
spec:
  type: ClusterIP
  ports:
  - port: 53
    name: tcp
    targetPort: tcp
    protocol: TCP
  - port: 53
    name: udp
    targetPort: udp
    protocol: UDP
  - port: 10000
    name: web
    targetPort: web
    protocol: TCP
  selector:
    app: bind

Then I've defined a master zone and created a forward dns record in it.
test.arman.local -> 192.168.230.216 

Next I have configured the configmap of coredns to use it:
kubectl edit configmap -n kube-system coredns

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    arman.local:53 {
        errors
        cache 30
        forward . 10.233.28.178
    }
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"Corefile":".:53 {\n    errors {\n    }\n    health {\n        lameduck 5s\n    }\n    ready\n    kubernetes cluster.local in-addr.arpa ip6.arpa {\n      pods insecure\n      fallthrough in-addr.arpa ip6.arpa\n    }\n    prometheus :9153\n    forward . /etc/resolv.conf {\n      prefer_udp\n      max_concurrent 1000\n    }\n    cache 30\n\n    loop\n    reload\n    loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists"},"name":"coredns","namespace":"kube-system"}}
  creationTimestamp: "2024-08-08T10:26:32Z"
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: coredns
  namespace: kube-system
  resourceVersion: "2135167"
  uid: 5457c863-3a34-4490-badb-036f4f0e1739

The service of BIND deployment is like:
kubectl get svc                 
NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
bind                          ClusterIP   10.233.28.178           53/TCP,53/UDP,10000/TCP      44h

The problem is it can't resolve test.arman.local
I've started to debug the problem I launched a busybox and used nslookup
kubectl run -it busybox --image=busybox:1.28 --restart=Never -- sh

Here is the content of /etc/resolv.conf of busybox
/ # cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.25.10
options ndots:5

I don't know what IP 169.254.25.10 is for because the service of dnsCore has different IP:
kubectl get svc -n kube-system 
NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
coredns   ClusterIP   10.233.0.3           53/UDP,53/TCP,9153/TCP   10d

In the busybox I can successfully resolve the test.arman.local using the following command:
nslookup test.arman.local 10.233.28.178
Server:    10.233.28.178
Address 1: 10.233.28.178

Name:      test.arman.local
Address 1: 192.168.230.216
/ # ^C
/ # cat /etc/resolv.conf 
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.25.10
options ndots:5

Or even use this:
nslookup test.arman.local 10.233.0.3 # This is for coredns
Server:    10.233.0.3
Address 1: 10.233.0.3 coredns.kube-system.svc.cluster.local

Name:      test.arman.local
Address 1: 192.168.230.216

We are using DNS/calico, for the sake of completeness I post the pods in kube-system:
 kubectl get pods -n kube-system 
NAME                                      READY   STATUS    RESTARTS        AGE
calico-kube-controllers-c7cc688f8-rb59q   1/1     Running   1 (7d21h ago)   10d
calico-node-5dglq                         1/1     Running   0               10d
calico-node-c592m                         1/1     Running   0               10d
calico-node-hbfqj                         1/1     Running   0               10d
calico-node-s6jrl                         1/1     Running   0               10d
coredns-857f949bf7-bprlf                  1/1     Running   0               65m
coredns-857f949bf7-r9p87                  1/1     Running   0               65m
dns-autoscaler-6ffb84bd6-fgdtz            1/1     Running   0               10d
kube-apiserver-node1                      1/1     Running   1               10d
kube-apiserver-node2                      1/1     Running   1               10d
kube-controller-manager-node1             1/1     Running   4 (7d21h ago)   10d
kube-controller-manager-node2             1/1     Running   7 (6d19h ago)   10d
kube-proxy-hb2wn                          1/1     Running   0               10d
kube-proxy-j5pfg                          1/1     Running   0               10d
kube-proxy-qjj4n                          1/1     Running   0               10d
kube-proxy-rp9mt                          1/1     Running   0               10d
kube-scheduler-node1                      1/1     Running   4 (6d19h ago)   10d
kube-scheduler-node2                      1/1     Running   6 (6d19h ago)   10d
nginx-proxy-node3                         1/1     Running   0               10d
nginx-proxy-node4                         1/1     Running   0               10d
nodelocaldns-6sztv                        1/1     Running   0               10d
nodelocaldns-bpt7s                        1/1     Running   0               10d
nodelocaldns-n4rsk                        1/1     Running   0               10d
nodelocaldns-n8q9w                        1/1     Running   0               10d

UPDATE: I found out it has something to do with nodelocaldns but I can not resolve the issue

**Tags:** kubernetes

> **Comments:**
> **username_1**: Probable duplicate of e.g. https://serverfault.com/questions/1009907/kubernetes-pod-etc-resolv-conf-has-the-wrong-nameserver  

---

## 2 answers

### Answer by username_0 on 2024-08-19. Score: 0 | Accepted answer

I managed to solve the issue by changing the configuration of nodelocaldns:
kubectl edit configmap -n kube-system nodelocaldns

change from
cluster.local:53 {
    errors
    cache {
        success 9984 30
        denial 9984 5
    }
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
    health 169.254.25.10:9254
}
in-addr.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
ip6.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
.:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . /etc/resolv.conf
    prometheus :9253
}

to this:
cluster.local:53 {
    errors
    cache {
        success 9984 30
        denial 9984 5
    }
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
    health 169.254.25.10:9254
}
in-addr.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
ip6.arpa:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}
.:53 {
    errors
    cache 30
    reload
    loop
    bind 169.254.25.10
    forward . 10.233.0.3 {
        force_tcp
    }
    prometheus :9253
}

look at the last block
I am curious what caused this issue in the first place?


---

### Answer by username_1 on 2024-08-19. Score: 0

In Kubernetes there is a convention that the cluster DNS service is on the `.10` IP address.

So it's expected that you'll configure the ClusterIP for CoreDNS to request that particular IP.

If not, you'll need to configure kubelet on each node to change what it injects into the `/etc/resolv.conf` file of most pods.

> **Comments:**
> **username_1**: Slightly more detail: https://serverfault.com/a/1158944/431900  

---

