<p>So after deleting all canal PODs and letting them recreate, everything started to work again as it should.
Seems like the recreation of those PODs somehow recreated routes and added the Worker node. Not sure why it didnt work right after a worker reboot, later when removing and re-joining this worker to the cluster, it got assigned a new CIDR, after canal POD recreation it started to work.</p>

CC BY-SA 4.0
2023-12-20T14:53:04.147Z
2023-12-20T14:53:04.147Z
Ghorio Fuitton
<p>Do you have vxlan in canal.yaml? Then change that to host-gw and see if it fixes the problem. flannel will communicate new routes to the kernel with netlink sockets then.</p>
<pre><code>  net-conf.json: |
    {
      &quot;Network&quot;: &quot;10.244.0.0/16&quot;,
      &quot;Backend&quot;: {
        &quot;Type&quot;: &quot;vxlan&quot; # CHANGE TO host-gw
      }
    }
</code></pre>
<p>More info here.</p>
<p><a href="https://devops.stackexchange.com/questions/18555/kubernetes-access-nodeport-fails-master-cannot-reach-workers/18557#18557">kubernetes access nodeport fails, master cannot reach workers</a></p>
<p>and on the flannel github.</p>
<p><a href="https://github.com/flannel-io/flannel" rel="nofollow noreferrer">https://github.com/flannel-io/flannel</a></p>
<p>If it still doesn't work, try adding the specific network interface flannel must work on. If you dont it will claim the first interface it finds.</p>
<pre><code> - name: kube-flannel
          image: quay.io/coreos/flannel:v0.9.1
          command: [ &quot;/opt/bin/flanneld&quot;, &quot;--ip-masq&quot;, &quot;--kube-subnet-mgr&quot; ]
</code></pre>
<p>add --iface=iface_you_want to &quot;command&quot;</p>

CC BY-SA 4.0
2023-12-14T09:57:31.673Z
2023-12-14T09:57:31.673Z
Serve Laurijssen
<p>Really looking for some advice and help, I have run out of ideas. Basically everything was working fine, 1 master and 3 workers. One Worker node was turned off to increase CPU and RAM resources and once turned back on, PODs running on this worker were not able to reach any other PODs and also DNS resolving was not working, so no external network connection also was working, e.g. google.com etc.
Here is a list of POD cidrs</p>
<pre><code>$ kubectl get nodes -o json | jq -r '.items[].spec.podCIDR' | sort
Master - 10.244.0.0/24
Worker1 - 10.244.1.0/24
Worker2 - 10.244.2.0/24
Worker3 - 10.244.5.0/24
</code></pre>
<p>The one which was restarted and stopped working is <strong>Worker3</strong>. When running dnsutils container on this node and trying some debuging</p>
<pre><code>root@dnsutils:/# nslookup kubernetes.default
;; connection timed out; no servers could be reached
</code></pre>
<p>Firewalld has been disabled and turned off.
Also when trying to ping all the networks from Worker3, only 10.244.1.0 and 10.244.5.0 (itself) is pinging</p>
<pre><code>root@dnsutils:/# ping 10.244.0.0
PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
^C
--- 10.244.0.0 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1043ms

root@dnsutils:/# ping 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
64 bytes from 10.244.1.0: icmp_seq=1 ttl=63 time=0.721 ms
64 bytes from 10.244.1.0: icmp_seq=2 ttl=63 time=0.947 ms
^C
--- 10.244.1.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1041ms
rtt min/avg/max/mdev = 0.721/0.834/0.947/0.113 ms
root@dnsutils:/# ping 10.244.2.0
PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
^C
--- 10.244.2.0 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1024ms

root@dnsutils:/# ping 10.244.5.0
PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
64 bytes from 10.244.5.0: icmp_seq=1 ttl=64 time=0.116 ms
64 bytes from 10.244.5.0: icmp_seq=2 ttl=64 time=0.142 ms
^C
--- 10.244.5.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1038ms
rtt min/avg/max/mdev = 0.116/0.129/0.142/0.013 ms
</code></pre>
<p>I believe the issue is because it cant reach coredns service/pods, which both are currently on 10.244.0.35 and 10.244.0.36 IPs</p>
<pre><code>NAME                                                                 READY   STATUS    RESTARTS         AGE     IP              NODE        NOMINATED NODE   READINESS GATES
calico-kube-controllers-7b564c7d85-rkc98                             1/1     Running   0                278d    10.244.0.30     Master       &lt;none&gt;           &lt;none&gt;
canal-5cffv                                                          2/2     Running   4 (378d ago)     630d    10.203.8.167    Worker3      &lt;none&gt;           &lt;none&gt;
canal-p822m                                                          2/2     Running   2 (474d ago)     630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
canal-scdrs                                                          2/2     Running   4 (351d ago)     630d    10.203.40.166   Worker2      &lt;none&gt;           &lt;none&gt;
canal-xkx7f                                                          2/2     Running   0                5h16m   10.203.8.174    Worker4      &lt;none&gt;           &lt;none&gt;
coredns-cc4c6799f-4tjbd                                              1/1     Running   0                11h     10.244.0.36     Master       &lt;none&gt;           &lt;none&gt;
coredns-cc4c6799f-bkq8n                                              1/1     Running   0                11h     10.244.0.35     Master       &lt;none&gt;           &lt;none&gt;
etcd-Master                                                          1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
kube-apiserver-Master                                                1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
kube-controller-manager-Master                                       1/1     Running   10 (265d ago)    392d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
kube-proxy-9n4wc                                                     1/1     Running   1 (474d ago)     630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
kube-proxy-rtccq                                                     1/1     Running   2 (351d ago)     630d    10.203.40.166   Worker2      &lt;none&gt;           &lt;none&gt;
kube-proxy-vtr55                                                     1/1     Running   0                5h16m   10.203.8.174    Worker4      &lt;none&gt;           &lt;none&gt;
kube-proxy-w9wkk                                                     1/1     Running   2 (378d ago)     630d    10.203.8.167    Worker3      &lt;none&gt;           &lt;none&gt;
kube-scheduler-Master                                                1/1     Running   326 (265d ago)   630d    10.203.8.165    Master       &lt;none&gt;           &lt;none&gt;
kube-state-metrics-8945855d-pvqqt                                    1/1     Running   0                139d    10.244.2.89     Worker3      &lt;none&gt;           &lt;none&gt;
metrics-server-7cf8b65d65-qpnzm                                      1/1     Running   2 (6h43m ago)    139d    10.244.2.83     Worker3      &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>Running dnsutils on Worker2 for example</p>
<pre><code>root@dnsutils2:/# nslookup kubernetes.default
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

root@dnsutils2:/# ping 10.244.0.0
PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
64 bytes from 10.244.0.0: icmp_seq=1 ttl=63 time=0.787 ms
64 bytes from 10.244.0.0: icmp_seq=2 ttl=63 time=0.835 ms
^C
--- 10.244.0.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1061ms
rtt min/avg/max/mdev = 0.787/0.811/0.835/0.024 ms
root@dnsutils2:/# ping 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
64 bytes from 10.244.1.0: icmp_seq=1 ttl=64 time=0.129 ms
64 bytes from 10.244.1.0: icmp_seq=2 ttl=64 time=0.152 ms
^C
--- 10.244.1.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.129/0.140/0.152/0.016 ms
root@dnsutils2:/# ping 10.244.2.0
PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
64 bytes from 10.244.2.0: icmp_seq=1 ttl=63 time=1.03 ms
64 bytes from 10.244.2.0: icmp_seq=2 ttl=63 time=0.933 ms
^C
--- 10.244.2.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.933/0.984/1.036/0.060 ms
root@dnsutils2:/# ping 10.244.5.0
PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
64 bytes from 10.244.5.0: icmp_seq=1 ttl=63 time=0.787 ms
64 bytes from 10.244.5.0: icmp_seq=2 ttl=63 time=0.995 ms
^C
--- 10.244.5.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.787/0.891/0.995/0.104 ms
</code></pre>
<p>Cluster was initialized running below command</p>
<pre><code>kubeadm init --apiserver-advertise-address=10.203.8.165 --pod-network-cidr=10.244.0.0/16
</code></pre>
<p>And for network cni plugin flannel-canal was used</p>
<pre><code>kubectl apply -f https://projectcalico.docs.tigera.io/manifests/canal.yaml
</code></pre>
<p>I would really appreciate any help on where to start debugging this.
Basically this Worker3 was working just fine before a restart and admins say that nothing was changed from network side or security groups etc. This is running on premises on VMware VMs.</p>
<p>I have also tried <code>kubeadm reset</code> and then <code>iptables -F &amp;&amp; iptables -t nat -F &amp;&amp; iptables -t mangle -F &amp;&amp; iptables -X</code> , rebooting the node and then <code>kubeadm join</code> but still the issue persists. No idea why this specific Worker3 wont ping anything besides 10.244.1.0 and 10.244.5.0 (itself)</p>
<p>I am confused about why wouldnt node be able to ping 10.244.2.0 network for example, but is able to ping 10.244.1.0 although there is iptables</p>
<pre><code>-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
</code></pre>
<p>and also routes</p>
<pre><code># route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.203.8.254    0.0.0.0         UG    100    0        0 ens192
10.203.8.0      0.0.0.0         255.255.255.0   U     100    0        0 ens192
10.244.0.0      10.244.0.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.1.0      10.244.1.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.2.0      10.244.2.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.5.2      0.0.0.0         255.255.255.255 UH    0      0        0 cali952544557fa
10.244.5.3      0.0.0.0         255.255.255.255 UH    0      0        0 calic15b507754a
10.244.5.4      0.0.0.0         255.255.255.255 UH    0      0        0 calif987de568c4
10.244.5.7      0.0.0.0         255.255.255.255 UH    0      0        0 cali969530ca43d
10.244.5.8      0.0.0.0         255.255.255.255 UH    0      0        0 calib3c61c3cba9
</code></pre>

<kubernetes>
Kubernetes worker node not able to reach POD network CIDR after reboot
Ghorio Fuitton
# Kubernetes worker node not able to reach POD network CIDR after reboot

(Asked by: username_0 on 2023-12-09)

Really looking for some advice and help, I have run out of ideas. Basically everything was working fine, 1 master and 3 workers. One Worker node was turned off to increase CPU and RAM resources and once turned back on, PODs running on this worker were not able to reach any other PODs and also DNS resolving was not working, so no external network connection also was working, e.g. google.com etc.
Here is a list of POD cidrs
$ kubectl get nodes -o json | jq -r '.items[].spec.podCIDR' | sort
Master - 10.244.0.0/24
Worker1 - 10.244.1.0/24
Worker2 - 10.244.2.0/24
Worker3 - 10.244.5.0/24

The one which was restarted and stopped working is Worker3. When running dnsutils container on this node and trying some debuging
root@dnsutils:/# nslookup kubernetes.default
;; connection timed out; no servers could be reached

Firewalld has been disabled and turned off.
Also when trying to ping all the networks from Worker3, only 10.244.1.0 and 10.244.5.0 (itself) is pinging
root@dnsutils:/# ping 10.244.0.0
PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
^C
--- 10.244.0.0 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1043ms

root@dnsutils:/# ping 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
64 bytes from 10.244.1.0: icmp_seq=1 ttl=63 time=0.721 ms
64 bytes from 10.244.1.0: icmp_seq=2 ttl=63 time=0.947 ms
^C
--- 10.244.1.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1041ms
rtt min/avg/max/mdev = 0.721/0.834/0.947/0.113 ms
root@dnsutils:/# ping 10.244.2.0
PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
^C
--- 10.244.2.0 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1024ms

root@dnsutils:/# ping 10.244.5.0
PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
64 bytes from 10.244.5.0: icmp_seq=1 ttl=64 time=0.116 ms
64 bytes from 10.244.5.0: icmp_seq=2 ttl=64 time=0.142 ms
^C
--- 10.244.5.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1038ms
rtt min/avg/max/mdev = 0.116/0.129/0.142/0.013 ms

I believe the issue is because it cant reach coredns service/pods, which both are currently on 10.244.0.35 and 10.244.0.36 IPs
NAME                                                                 READY   STATUS    RESTARTS         AGE     IP              NODE        NOMINATED NODE   READINESS GATES
calico-kube-controllers-7b564c7d85-rkc98                             1/1     Running   0                278d    10.244.0.30     Master                  
canal-5cffv                                                          2/2     Running   4 (378d ago)     630d    10.203.8.167    Worker3                 
canal-p822m                                                          2/2     Running   2 (474d ago)     630d    10.203.8.165    Master                  
canal-scdrs                                                          2/2     Running   4 (351d ago)     630d    10.203.40.166   Worker2                 
canal-xkx7f                                                          2/2     Running   0                5h16m   10.203.8.174    Worker4                 
coredns-cc4c6799f-4tjbd                                              1/1     Running   0                11h     10.244.0.36     Master                  
coredns-cc4c6799f-bkq8n                                              1/1     Running   0                11h     10.244.0.35     Master                  
etcd-Master                                                          1/1     Running   6 (474d ago)     630d    10.203.8.165    Master                  
kube-apiserver-Master                                                1/1     Running   6 (474d ago)     630d    10.203.8.165    Master                  
kube-controller-manager-Master                                       1/1     Running   10 (265d ago)    392d    10.203.8.165    Master                  
kube-proxy-9n4wc                                                     1/1     Running   1 (474d ago)     630d    10.203.8.165    Master                  
kube-proxy-rtccq                                                     1/1     Running   2 (351d ago)     630d    10.203.40.166   Worker2                 
kube-proxy-vtr55                                                     1/1     Running   0                5h16m   10.203.8.174    Worker4                 
kube-proxy-w9wkk                                                     1/1     Running   2 (378d ago)     630d    10.203.8.167    Worker3                 
kube-scheduler-Master                                                1/1     Running   326 (265d ago)   630d    10.203.8.165    Master                  
kube-state-metrics-8945855d-pvqqt                                    1/1     Running   0                139d    10.244.2.89     Worker3                 
metrics-server-7cf8b65d65-qpnzm                                      1/1     Running   2 (6h43m ago)    139d    10.244.2.83     Worker3                 

Running dnsutils on Worker2 for example
root@dnsutils2:/# nslookup kubernetes.default
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

root@dnsutils2:/# ping 10.244.0.0
PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
64 bytes from 10.244.0.0: icmp_seq=1 ttl=63 time=0.787 ms
64 bytes from 10.244.0.0: icmp_seq=2 ttl=63 time=0.835 ms
^C
--- 10.244.0.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1061ms
rtt min/avg/max/mdev = 0.787/0.811/0.835/0.024 ms
root@dnsutils2:/# ping 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
64 bytes from 10.244.1.0: icmp_seq=1 ttl=64 time=0.129 ms
64 bytes from 10.244.1.0: icmp_seq=2 ttl=64 time=0.152 ms
^C
--- 10.244.1.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.129/0.140/0.152/0.016 ms
root@dnsutils2:/# ping 10.244.2.0
PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
64 bytes from 10.244.2.0: icmp_seq=1 ttl=63 time=1.03 ms
64 bytes from 10.244.2.0: icmp_seq=2 ttl=63 time=0.933 ms
^C
--- 10.244.2.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.933/0.984/1.036/0.060 ms
root@dnsutils2:/# ping 10.244.5.0
PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
64 bytes from 10.244.5.0: icmp_seq=1 ttl=63 time=0.787 ms
64 bytes from 10.244.5.0: icmp_seq=2 ttl=63 time=0.995 ms
^C
--- 10.244.5.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.787/0.891/0.995/0.104 ms

Cluster was initialized running below command
kubeadm init --apiserver-advertise-address=10.203.8.165 --pod-network-cidr=10.244.0.0/16

And for network cni plugin flannel-canal was used
kubectl apply -f https://projectcalico.docs.tigera.io/manifests/canal.yaml

I would really appreciate any help on where to start debugging this.
Basically this Worker3 was working just fine before a restart and admins say that nothing was changed from network side or security groups etc. This is running on premises on VMware VMs.
I have also tried kubeadm reset and then iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X , rebooting the node and then kubeadm join but still the issue persists. No idea why this specific Worker3 wont ping anything besides 10.244.1.0 and 10.244.5.0 (itself)
I am confused about why wouldnt node be able to ping 10.244.2.0 network for example, but is able to ping 10.244.1.0 although there is iptables
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT

and also routes
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.203.8.254    0.0.0.0         UG    100    0        0 ens192
10.203.8.0      0.0.0.0         255.255.255.0   U     100    0        0 ens192
10.244.0.0      10.244.0.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.1.0      10.244.1.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.2.0      10.244.2.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.5.2      0.0.0.0         255.255.255.255 UH    0      0        0 cali952544557fa
10.244.5.3      0.0.0.0         255.255.255.255 UH    0      0        0 calic15b507754a
10.244.5.4      0.0.0.0         255.255.255.255 UH    0      0        0 calif987de568c4
10.244.5.7      0.0.0.0         255.255.255.255 UH    0      0        0 cali969530ca43d
10.244.5.8      0.0.0.0         255.255.255.255 UH    0      0        0 calib3c61c3cba9

**Tags:** kubernetes


---

## 2 answers

### Answer by username_0 on 2023-12-20. Score: 0 | Accepted answer

So after deleting all canal PODs and letting them recreate, everything started to work again as it should.
Seems like the recreation of those PODs somehow recreated routes and added the Worker node. Not sure why it didnt work right after a worker reboot, later when removing and re-joining this worker to the cluster, it got assigned a new CIDR, after canal POD recreation it started to work.


---

### Answer by username_1 on 2023-12-14. Score: 0

Do you have vxlan in canal.yaml? Then change that to host-gw and see if it fixes the problem. flannel will communicate new routes to the kernel with netlink sockets then.
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan" # CHANGE TO host-gw
      }
    }

More info here.
kubernetes access nodeport fails, master cannot reach workers
and on the flannel github.
https://github.com/flannel-io/flannel
If it still doesn't work, try adding the specific network interface flannel must work on. If you dont it will claim the first interface it finds.
 - name: kube-flannel
          image: quay.io/coreos/flannel:v0.9.1
          command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]

add --iface=iface_you_want to "command"


---

