<p>I'm running crio 1.17.0-rc1 with kubernetes 1.17.1 on RHEL7/CENTOS7.  One thing I had to do was build crio 1.17 myself because there was no package available and I read the version should match (crio 1.17 and k8 1.17).</p>

<p>I am using systemd as the cgroup driver. I'm not sure what your issue might be, but my setup goes like this:</p>

<pre><code>cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///var/run/crio/crio.sock --cloud-provider=external
</code></pre>

<p>My kubeadm init line is (I'm using an env variable to set the control plain endpoint):</p>

<pre><code>kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint {{ lookup('env','master_lb_ip_address') }}:6443 --upload-certs &gt;&gt; $HOME/log/init-cluster.log
</code></pre>

<p>I also edited /etc/crio/crio.conf to set systemd as the cgroup manager.</p>

<pre><code>cgroup_manager = "systemd"
</code></pre>

<p>Also make sure you are enabling the modules / sysconfig options specified on <a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o" rel="nofollow noreferrer">this page</a>.</p>

<pre><code>modprobe overlay
modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat &gt; /etc/sysctl.d/99-kubernetes-cri.conf &lt;&lt;EOF
net.bridge.bridge-nf-call-iptables  = 1
net.ipv4.ip_forward                 = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

sysctl --system
</code></pre>

CC BY-SA 4.0
2020-02-11T11:46:39.013Z
Thanks for your suggestions. Please review the question as I've updated it with new info and also with the outputs of running kubeadm init with your parameters. Still no luck :/
carrotcakeslayer
CC BY-SA 4.0
2020-02-11T12:34:40.283Z
What's your cri-o version?  Can you show output from `crictl version` and `crictl info`?
Luke
CC BY-SA 4.0
2020-02-12T11:48:12.787Z
Hi LArntz. I've just added the info to the original question. Thank you!
carrotcakeslayer
CC BY-SA 4.0
2020-02-12T12:05:32.777Z
OK. I think this might be your problem....

`crictl version -> ["Version: 0.1.0","RuntimeName: cri-o","RuntimeVersion: 1.15.1-2.el7","RuntimeApiVersion: v1alpha1"]`

See the [compatibilty matrix on the cri-o github](https://github.com/cri-o/cri-o).  You need to be running a version of cri-o that matches the kubernetes version. 

If there's not binary version available for your OS it's pretty easy [to build](https://github.com/cri-o/cri-o/blob/master/tutorials/setup.md) (but I've heard of issues with redhat/centos 8).
Luke
CC BY-SA 4.0
2020-02-12T12:08:38.910Z
Makes sense and matches your original answer. I overviewed this point as it is not mentioned in the "getting started" kubernetes doc. I'll try to build the package and comment here the output. Thank you!
carrotcakeslayer
CC BY-SA 4.0
2020-02-12T12:11:23.810Z
That's a good point. If their docs are opensource a pull request might be helpful to others.
Luke
CC BY-SA 4.0
2020-02-12T15:13:12.973Z
That was it!!! Also, in cri-o github page it says that kubelet needs to be same version as cri-o (not mentioned in kubernetes doc for kubeadm install where only kubeadm and kubelet match is mentioned). I was not able to compile v1.17, so I just shortcut to run all in v1.15 (latest available in repo). Thank you very much!
carrotcakeslayer
CC BY-SA 4.0
2020-02-12T17:07:50.080Z
No problem! Happy your issue is resolved. 

I submitted a pull request for the kubernetes docs also so hopefully that will be updated soon.
Luke
CC BY-SA 4.0
2020-02-10T19:24:08.633Z
2020-02-10T19:24:08.633Z
Luke
<p>I've been following this doc to deploy kubernetes <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/" rel="nofollow noreferrer">https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/</a></p>

<p>After launching the "kubeadm init" command and during its excecution, this is the output:</p>

<blockquote>
  <p>[kubelet-check] The HTTP call equal to 'curl -sSL <a href="http://localhost:10248/healthz" rel="nofollow noreferrer">http://localhost:10248/healthz</a>' failed with error: Get <a href="http://localhost:10248/healthz" rel="nofollow noreferrer">http://localhost:10248/healthz</a>: dial tcp 127.0.0.1:10248: connect: connection refused.
  Unfortunately, an error has occurred:
          timed out waiting for the condition
  This error is likely caused by:
          - The kubelet is not running
          - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
  If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
          - 'systemctl status kubelet'
          - 'journalctl -xeu kubelet'
  Additionally, a control plane component may have crashed or exited when started by the container runtime.
  To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
  Here is one example how you may list all Kubernetes containers running in docker:
          - 'docker ps -a | grep kube | grep -v pause'
          Once you have found the failing container, you can inspect its logs with:
          - 'docker logs CONTAINERID'
  error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
  To see the stack trace of this error execute with --v=5 or higher</p>
</blockquote>

<p>It runs fine if I install docker and switch the runtime to docker instead of "remote" and "systemd".</p>

<p>I've tried several things, but this is the content of my last try:</p>

<blockquote>
  <p>cat etc/sysconfig/kubelet</p>
  
  <p>KUBELET_ARGS="--container-runtime=remote
  --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=10m --cluster_domain=topota.madre"</p>
  
  <p>systemctl cat kubelet</p>
  
  <p># /usr/lib/systemd/system/kubelet.service [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=<a href="https://kubernetes.io/docs/" rel="nofollow noreferrer">https://kubernetes.io/docs/</a></p>
  
  <p>[Service] ExecStart=/usr/bin/kubelet Restart=always
  StartLimitInterval=0 RestartSec=10</p>
  
  <p>[Install] WantedBy=multi-user.target</p>
  
  <p># /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
  # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service]
  Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
  --kubeconfig=/etc/ Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
  # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS var
  EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
  # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user shoul
  # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sour
  EnvironmentFile=-/etc/sysconfig/kubelet ExecStart=
  ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS
  $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS</p>
  
  <p>cat /var/lib/kubelet/kubeadm-flags.env</p>
  
  <p>is empty right now</p>
</blockquote>

<p>Cri-O service ir running ok and Docker is not installed.</p>

<ul>
<li>kubelet version --short -> Client Version: v1.17.2</li>
<li>kubeadm version -o short -> v1.17.2</li>
<li>cat /etc/redhat-release -> CentOS Linux release 7.7.1908 (Core)</li>
<li>crictl version -> ["Version:  0.1.0","RuntimeName: cri-o","RuntimeVersion:  1.15.1-2.el7","RuntimeApiVersion:  v1alpha1"]</li>
<li>crictl info -> {"status": {"conditions": [{"type": "RuntimeReady","status": true,"reason": "","message": ""},{"type": NetworkReady","status": true,"reason": "","message": ""}]}}</li>
<li>grep -i cgroup_manager /etc/crio/crio.conf -> cgroup_manager = "systemd"</li>
<li>lsmod | grep -i "overlay\|br_netfilter" -> ["br_netfilter","overlay"] (both ok).</li>
<li>Other "extra args tested" (/etc/sysconfig/kubelet)</li>
</ul>

<blockquote>
  <p>KUBELET_EXTRA_ARGS=--cgroup-driver=systemd
  --container-runtime-endpoint=unix:///var/run/crio/crio.sock</p>
</blockquote>

<ul>
<li>Full output of some commands:
-> "kube admin init --upload-certs" -> <a href="https://pastebin.com/isbAynEg" rel="nofollow noreferrer">https://pastebin.com/isbAynEg</a>
-> "while true ; do systemctl status kubelet ; sleep 1 ; done" -> <a href="https://pastebin.com/9pgtDMNu" rel="nofollow noreferrer">https://pastebin.com/9pgtDMNu</a>
-> "cat /etc/sysctl.d/99-kubernetes-cri.conf" -> ["net.bridge.bridge-nf-call-iptables = 1","net.ipv4.ip_forward = 1","net.bridge.bridge-nf-call-ip6tables = 1"]
-> "journalctl -xeu kubelet" -> <a href="https://pastebin.com/qreJyhsu" rel="nofollow noreferrer">https://pastebin.com/qreJyhsu</a> (this one has some things like "aws credentials" which i dont understand and not sure if are the reason or not for the errors).</li>
</ul>

<p>Any help please?</p>

CC BY-SA 4.0
2020-02-10T19:28:21.667Z
What's the output when you follow instructions bellow:

The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet'
Vader
CC BY-SA 4.0
2020-02-11T11:45:48.343Z
I just added the links to pastebin with those outputs. I did not see anything conclusive in them, but there are some interesting lines that I may be overseeing.
carrotcakeslayer
CC BY-SA 4.0
2020-02-11T21:59:39.167Z
What distribution are you using?
Matt
CC BY-SA 4.0
2020-02-12T11:47:45.023Z
I've just added the info to the original question. Thx for your help!
carrotcakeslayer
<kubernetes><kubeadm>
kubeadm init fails when using cri-o
carrotcakeslayer
# kubeadm init fails when using cri-o

(Asked by: username_0 on 2020-02-10)

I've been following this doc to deploy kubernetes <https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/>

After launching the "kubeadm init" command and during its excecution, this is the output:

> [kubelet-check] The HTTP call equal to 'curl -sSL <http://localhost:10248/healthz>' failed with error: Get <http://localhost:10248/healthz>: dial tcp 127.0.0.1:10248: connect: connection refused.
> Unfortunately, an error has occurred:
> timed out waiting for the condition
> This error is likely caused by:
> - The kubelet is not running
> - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
> If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
> - 'systemctl status kubelet'
> - 'journalctl -xeu kubelet'
> Additionally, a control plane component may have crashed or exited when started by the container runtime.
> To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
> Here is one example how you may list all Kubernetes containers running in docker:
> - 'docker ps -a | grep kube | grep -v pause'
> Once you have found the failing container, you can inspect its logs with:
> - 'docker logs CONTAINERID'
> error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
> To see the stack trace of this error execute with --v=5 or higher

It runs fine if I install docker and switch the runtime to docker instead of "remote" and "systemd".

I've tried several things, but this is the content of my last try:

> cat etc/sysconfig/kubelet
>
> KUBELET\_ARGS="--container-runtime=remote
> --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=10m --cluster\_domain=topota.madre"
>
> systemctl cat kubelet
>
> # /usr/lib/systemd/system/kubelet.service [Unit] Description=kubelet: The Kubernetes Node Agent Documentation=<https://kubernetes.io/docs/>
>
> [Service] ExecStart=/usr/bin/kubelet Restart=always
> StartLimitInterval=0 RestartSec=10
>
> [Install] WantedBy=multi-user.target
>
> # /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
> # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service]
> Environment="KUBELET\_KUBECONFIG\_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
> --kubeconfig=/etc/ Environment="KUBELET\_CONFIG\_ARGS=--config=/var/lib/kubelet/config.yaml"
> # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET\_KUBEADM\_ARGS var
> EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
> # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user shoul
> # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET\_EXTRA\_ARGS should be sour
> EnvironmentFile=-/etc/sysconfig/kubelet ExecStart=
> ExecStart=/usr/bin/kubelet $KUBELET\_KUBECONFIG\_ARGS
> $KUBELET\_CONFIG\_ARGS $KUBELET\_KUBEADM\_ARGS $KUBELET\_EXTRA\_ARGS
>
> cat /var/lib/kubelet/kubeadm-flags.env
>
> is empty right now

Cri-O service ir running ok and Docker is not installed.

* kubelet version --short -> Client Version: v1.17.2
* kubeadm version -o short -> v1.17.2
* cat /etc/redhat-release -> CentOS Linux release 7.7.1908 (Core)
* crictl version -> ["Version: 0.1.0","RuntimeName: cri-o","RuntimeVersion: 1.15.1-2.el7","RuntimeApiVersion: v1alpha1"]
* crictl info -> {"status": {"conditions": [{"type": "RuntimeReady","status": true,"reason": "","message": ""},{"type": NetworkReady","status": true,"reason": "","message": ""}]}}
* grep -i cgroup\_manager /etc/crio/crio.conf -> cgroup\_manager = "systemd"
* lsmod | grep -i "overlay\|br\_netfilter" -> ["br\_netfilter","overlay"] (both ok).
* Other "extra args tested" (/etc/sysconfig/kubelet)

> KUBELET\_EXTRA\_ARGS=--cgroup-driver=systemd
> --container-runtime-endpoint=unix:///var/run/crio/crio.sock

* Full output of some commands:
  -> "kube admin init --upload-certs" -> <https://pastebin.com/isbAynEg>
  -> "while true ; do systemctl status kubelet ; sleep 1 ; done" -> <https://pastebin.com/9pgtDMNu>
  -> "cat /etc/sysctl.d/99-kubernetes-cri.conf" -> ["net.bridge.bridge-nf-call-iptables = 1","net.ipv4.ip\_forward = 1","net.bridge.bridge-nf-call-ip6tables = 1"]
  -> "journalctl -xeu kubelet" -> <https://pastebin.com/qreJyhsu> (this one has some things like "aws credentials" which i dont understand and not sure if are the reason or not for the errors).

Any help please?

**Tags:** kubernetes, kubeadm

> **Comments:**
> **username_1**: What's the output when you follow instructions bellow:  The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet'  
> **username_0**: I just added the links to pastebin with those outputs. I did not see anything conclusive in them, but there are some interesting lines that I may be overseeing.  
> **username_2**: What distribution are you using?  
> **username_0**: I've just added the info to the original question. Thx for your help!  

---

## 1 answers

### Answer by username_3 on 2020-02-10. Score: 2 | Accepted answer

I'm running crio 1.17.0-rc1 with kubernetes 1.17.1 on RHEL7/CENTOS7.  One thing I had to do was build crio 1.17 myself because there was no package available and I read the version should match (crio 1.17 and k8 1.17).
I am using systemd as the cgroup driver. I'm not sure what your issue might be, but my setup goes like this:
cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd --container-runtime-endpoint=unix:///var/run/crio/crio.sock --cloud-provider=external

My kubeadm init line is (I'm using an env variable to set the control plain endpoint):
kubeadm init --pod-network-cidr=10.244.0.0/16 --control-plane-endpoint {{ lookup('env','master_lb_ip_address') }}:6443 --upload-certs >> $HOME/log/init-cluster.log

I also edited /etc/crio/crio.conf to set systemd as the cgroup manager.
cgroup_manager = "systemd"

Also make sure you are enabling the modules / sysconfig options specified on this page.
modprobe overlay
modprobe br_netfilter

# Setup required sysctl params, these persist across reboots.
cat > /etc/sysctl.d/99-kubernetes-cri.conf <

> **Comments:**
> **username_0**: Thanks for your suggestions. Please review the question as I've updated it with new info and also with the outputs of running kubeadm init with your parameters. Still no luck :/  
> **username_3**: What's your cri-o version?  Can you show output from `crictl version` and `crictl info`?  
> **username_0**: Hi LArntz. I've just added the info to the original question. Thank you!  
> **username_3**: OK. I think this might be your problem....  `crictl version -> ["Version: 0.1.0","RuntimeName: cri-o","RuntimeVersion: 1.15.1-2.el7","RuntimeApiVersion: v1alpha1"]`  See the [compatibilty matrix on the cri-o github](https://github.com/cri-o/cri-o).  You need to be running a version of cri-o that matches the kubernetes version.   If there's not binary version available for your OS it's pretty easy [to build](https://github.com/cri-o/cri-o/blob/master/tutorials/setup.md) (but I've heard of issues with redhat/centos 8).  
> **username_0**: Makes sense and matches your original answer. I overviewed this point as it is not mentioned in the "getting started" kubernetes doc. I'll try to build the package and comment here the output. Thank you!  
> **username_3**: That's a good point. If their docs are opensource a pull request might be helpful to others.  
> **username_0**: That was it!!! Also, in cri-o github page it says that kubelet needs to be same version as cri-o (not mentioned in kubernetes doc for kubeadm install where only kubeadm and kubelet match is mentioned). I was not able to compile v1.17, so I just shortcut to run all in v1.15 (latest available in repo). Thank you very much!  
> **username_3**: No problem! Happy your issue is resolved.   I submitted a pull request for the kubernetes docs also so hopefully that will be updated soon.  

---

