kubeasz错误问题(failed to run Kubelet: misconfiguration: kubelet cgroup driver)

100 阅读 0 评论 66 点赞

我是靠谱客的博主坚定长颈鹿，最近开发中收集的这篇文章主要介绍kubeasz错误问题(failed to run Kubelet: misconfiguration: kubelet cgroup driver)，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

运行环境

Ubuntu 20.04
DOCKER 19.03.15
KUBEASZ 3.1.0
K8S_BIN 1.21.0

1 错误1

1.1 事件还原

由于使用 kubeasz（地址：https://github.com/easzlab/kubeasz）
部署二进制的 k8s ，为了便于操作，我提前将 k8s-master1，k8s-master2，k8s-master3，k8s-node1，k8s-node2，k8s-node3 都安装了 docker，这样在执行 ezctl setup k8s-01 03 的时候，发现每台机器都安装了 docker 就不会再执行安装 docker 的操作了，相应的也不会再修改 /etc/docker/daemon.json 文件，直接导致 docker 运行的命名空间和默认的命名空间不一致，所以无法启动 kubelet 服务

执行 [root@k8s-master1 kubeasz]# ./ezctl setup k8s-01 04

报错如下：

TASK [kube-node : 轮询等待kubelet启动] *******************************************************************************************************
changed: [172.18.60.202]
changed: [172.18.60.201]
FAILED - RETRYING: 轮询等待node达到Ready状态 (8 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (8 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (7 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (7 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (6 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (6 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (5 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (5 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (4 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (4 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (3 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (3 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (2 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (2 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (1 retries left).
FAILED - RETRYING: 轮询等待node达到Ready状态 (1 retries left).
TASK [kube-node : 轮询等待node达到Ready状态] ***************************************************************************************************
fatal: [172.18.60.201]: FAILED! => {"attempts": 8, "changed": true, "cmd": "/usr/bin/kubectl get node 172.18.60.201|awk 'NR>1{print $2}'", "delta": "0:00:00.263088", "end": "2021-09-01 12:26:16.691810", "rc": 0, "start": "2021-09-01 12:26:16.428722", "stderr": "Error from server (NotFound): nodes "172.18.60.201" not found", "stderr_lines": ["Error from server (NotFound): nodes "172.18.60.201" not found"], "stdout": "", "stdout_lines": []}
fatal: [172.18.60.202]: FAILED! => {"attempts": 8, "changed": true, "cmd": "/usr/bin/kubectl get node 172.18.60.202|awk 'NR>1{print $2}'", "delta": "0:00:00.283975", "end": "2021-09-01 12:26:17.056414", "rc": 0, "start": "2021-09-01 12:26:16.772439", "stderr": "Error from server (NotFound): nodes "172.18.60.202" not found", "stderr_lines": ["Error from server (NotFound): nodes "172.18.60.202" not found"], "stdout": "", "stdout_lines": []}
PLAY RECAP *****************************************************************************************************************************
172.18.60.201
: ok=53
changed=28
unreachable=0
failed=1
skipped=1
rescued=0
ignored=0
172.18.60.202
: ok=50
changed=26
unreachable=0
failed=1
skipped=0
rescued=0
ignored=0
[root@k8s-master1 kubeasz]#

使用 journalctl -u kubelet 查看日志

为了便于查看，我们可以将日志重定向到一个文件中

journalctl -u kubelet > /tmp/kubelet.log

发现错误

Sep 02 08:52:33 k8s-master1.example.com kubelet[612230]: E0902 08:52:33.526931
612230 server.go:292] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs""
Sep 02 08:52:33 k8s-master1.example.com systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 02 08:52:33 k8s-master1.example.com systemd[1]: kubelet.service: Failed with result 'exit-code'.

1.2 解决办法

[root@k8s-master1 ~]# cat /etc/docker/daemon.json
{
"data-root": "/var/lib/docker",
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": [
"https://docker.mirrors.ustc.edu.cn",
"http://hub-mirror.c.163.com"
],
"insecure-registries": ["127.0.0.1/8","172.18.8.214","172.18.8.215"],
"max-concurrent-downloads": 10,
"live-restore": true,
"log-driver": "json-file",
"log-level": "warn",
"log-opts": {
"max-size": "50m",
"max-file": "1"
},
"storage-driver": "overlay2"
}
[root@k8s-master1 ~]#
[root@k8s-master1 ~]# docker info|grep "Cgroup Driver"
WARNING: No swap limit support
Cgroup Driver: systemd
[root@k8s-master1 ~]#

注意：Ubuntu18 的 /etc/docker/daemon.json 文件如下

root@k8s-master-1:~# docker info|grep "Cgroup Driver"
Cgroup Driver: cgroupfs
WARNING: No swap limit support
root@k8s-master-1:~# cat /etc/docker/daemon.json
{
"data-root": "/var/lib/docker",
"exec-opts": ["native.cgroupdriver=cgroupfs"],
"insecure-registries": ["127.0.0.1/8","172.18.8.214"],
"max-concurrent-downloads": 10,
"live-restore": true,
"log-driver": "json-file",
"log-level": "warn",
"log-opts": {
"max-size": "50m",
"max-file": "1"
},
"storage-driver": "overlay2"
}

2 错误2

Sep 02 16:31:17 k8s-master2.com kubelet[87623]: W0902 16:31:17.259389
87623 cni.go:204] Error validating CNI config list {"cniVersion":"0.3.1","name":"mynet","plugins":[{"bridge":"mynet0","cniVersion":"0.3.1","hairpinMode":true,"ipMasq":true,"ipam":{"subnet":"10.200.0.0/16","type":"host-local"},"isDefaultGateway":true,"name":"mynet","type":"bridge"}]}: [failed to find plugin "bridge" in path [/opt/cni/bin]]
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: W0902 16:31:17.260166
87623 cni.go:204] Error validating CNI config list {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"name": "cbr0",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"cniVersion": "0.3.1",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"plugins": [
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
{
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"type": "flannel",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"delegate": {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"hairpinMode": true,
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"isDefaultGateway": true
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
}
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
},
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
{
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"type": "portmap",
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"capabilities": {
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
"portMappings": true
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
}
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
}
Sep 02 16:31:17 k8s-master2.com kubelet[87623]:
]
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: }
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: : [failed to find plugin "flannel" in path [/opt/cni/bin] failed to find plugin "portmap" in path [/opt/cni/bin]]
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: W0902 16:31:17.260197
87623 cni.go:239] Unable to update cni config: no valid networks found in /etc/cni/net.d
Sep 02 16:31:17 k8s-master2.com kubelet[87623]: E0902 16:31:17.328048
87623 kubelet.go:2243] node "k8s-master2.com" not found