<p>I had set up a Kubernetes cluster on four Raspberry Pis for toy projects awhile back, and I originally set things up using Docker. But ever since Kubernetes decided to drop support for Docker I decided to re-do the cluster using ContainerD instead. On top of that, I wiped the SD cards and installed the x64 version of Raspberry Pi OS (instead of the 32-bit version which I was using previously). I've slowly been piecing together a runbook of how to install things but have encountered a snag: five minutes after calling <code>kubeadm init</code> on the master node, the cluster stops working. For reference, the internal IP of my master node is 192.168.1.194. Here's the full command I use to initialize things:</p>
<pre><code>sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
    --token-ttl=0 --apiserver-advertise-address=192.168.1.194
</code></pre>
<p>After running that command, and subsequently copying the /etc/kubernetes/admin.conf file to ~/.kube/config, I can run the following command:</p>
<pre><code>$ kubectl get nodes

NAME           STATUS     ROLES                  AGE     VERSION
k8s-master-1   NotReady   control-plane,master   3m36s   v1.23.4
</code></pre>
<p>And it will continue to show a <code>NotReady</code> status for about 5 minutes, after which point the same command yields a very different result:</p>
<pre><code>$ kubectl get nodes

The connection to the server 192.168.1.194:6443 was refused - did you specify the right host or port?
</code></pre>
<p>I'm not sure why this is happening, but it is very consistent. I have tried a few times now to <code>kubeadm reset</code> and then <code>kubeadm init</code> again, and the connection timeout always happens at the 5-minute mark. So the last time I tried to do this, I decided to tail all the log files under /var/log/containers/. After the 5-minute mark, it is repeatedly logging some variation of a connection error to 127.0.0.1:2379. For example:</p>
<blockquote>
<p>2022-03-09T19:30:29.307156643-06:00 stderr F W0310 01:30:29.306871       1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1  0 }. Err: connection error: desc = &quot;transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused&quot;. Reconnecting...</p>
</blockquote>
<p>But prior to the 5-minute mark, it looks like a bunch of services start shutting down right at the end, followed by what looks like maybe etcd shuts down as well. I've uploaded the full logs from the time <code>kubeadm init</code> runs, up until the dreaded 5-minute mark, <a href="https://gist.github.com/soapergem/fd71b313473aa366e5bb44acdaccaab4" rel="nofollow noreferrer">as a Gist</a>.</p>
<p>Why won't Kubernetes start on my Pi? What am I missing?</p>

CC BY-SA 4.0
2022-03-16T16:28:32.560Z
does all the ports are open? could you do test all ports like  `telnet localhost 2379`.
Ref docs for all the ports: https://kubernetes.io/docs/reference/ports-and-protocols/
Vinod
CC BY-SA 4.0
2022-03-17T02:41:14.080Z
Yes, all ports are open. The telnet command succeeds during that first 5 minute window of initialization, while etcd is active.
soapergem
CC BY-SA 4.0
2022-03-17T07:02:04.703Z
if all steps are correct and it is failing then what i can think of is `kubeadm join` is not able to bring this node in cluster. there could be multiple reason for that. may be you could try to specify the `--control-plane-endpoint` while configuring the cluster which is basically your cluster endpoint. 
Ref: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#:~:text=While%20%2D%2Dapiserver%2Dadvertise,supported%20by%20kubeadm
Vinod
CC BY-SA 4.0
2022-03-21T02:52:44.587Z
Again, this is not one of the nodes but the master, so `kubeadm join` is not used.
soapergem
<kubernetes><containerd><raspberry-pi-os><etcd>
Kubernetes cluster on Raspberry Pi stops working 5 minutes after init
soapergem
# Kubernetes cluster on Raspberry Pi stops working 5 minutes after init

(Asked by: username_0 on 2022-03-10)

I had set up a Kubernetes cluster on four Raspberry Pis for toy projects awhile back, and I originally set things up using Docker. But ever since Kubernetes decided to drop support for Docker I decided to re-do the cluster using ContainerD instead. On top of that, I wiped the SD cards and installed the x64 version of Raspberry Pi OS (instead of the 32-bit version which I was using previously). I've slowly been piecing together a runbook of how to install things but have encountered a snag: five minutes after calling kubeadm init on the master node, the cluster stops working. For reference, the internal IP of my master node is 192.168.1.194. Here's the full command I use to initialize things:
sudo kubeadm init --pod-network-cidr=10.244.0.0/16 \
    --token-ttl=0 --apiserver-advertise-address=192.168.1.194

After running that command, and subsequently copying the /etc/kubernetes/admin.conf file to ~/.kube/config, I can run the following command:
$ kubectl get nodes

NAME           STATUS     ROLES                  AGE     VERSION
k8s-master-1   NotReady   control-plane,master   3m36s   v1.23.4

And it will continue to show a NotReady status for about 5 minutes, after which point the same command yields a very different result:
$ kubectl get nodes

The connection to the server 192.168.1.194:6443 was refused - did you specify the right host or port?

I'm not sure why this is happening, but it is very consistent. I have tried a few times now to kubeadm reset and then kubeadm init again, and the connection timeout always happens at the 5-minute mark. So the last time I tried to do this, I decided to tail all the log files under /var/log/containers/. After the 5-minute mark, it is repeatedly logging some variation of a connection error to 127.0.0.1:2379. For example:

2022-03-09T19:30:29.307156643-06:00 stderr F W0310 01:30:29.306871       1 clientconn.go:1331] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1  0 }. Err: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...

But prior to the 5-minute mark, it looks like a bunch of services start shutting down right at the end, followed by what looks like maybe etcd shuts down as well. I've uploaded the full logs from the time kubeadm init runs, up until the dreaded 5-minute mark, as a Gist.
Why won't Kubernetes start on my Pi? What am I missing?

**Tags:** kubernetes, containerd, raspberry-pi-os, etcd

> **Comments:**
> **username_1**: does all the ports are open? could you do test all ports like  `telnet localhost 2379`. Ref docs for all the ports: https://kubernetes.io/docs/reference/ports-and-protocols/  
> **username_0**: Yes, all ports are open. The telnet command succeeds during that first 5 minute window of initialization, while etcd is active.  
> **username_1**: if all steps are correct and it is failing then what i can think of is `kubeadm join` is not able to bring this node in cluster. there could be multiple reason for that. may be you could try to specify the `--control-plane-endpoint` while configuring the cluster which is basically your cluster endpoint.  Ref: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#:~:text=While%20%2D%2Dapiserver%2Dadvertise,supported%20by%20kubeadm  
> **username_0**: Again, this is not one of the nodes but the master, so `kubeadm join` is not used.  

---

## 0 answers

