Kubernetes：node处于 NotReady 状态，恢复办法

62 阅读 0 评论 41 点赞

我是靠谱客的博主清脆夏天，最近开发中收集的这篇文章主要介绍Kubernetes：node处于 NotReady 状态，恢复办法，觉得挺不错的，现在分享给大家，希望可以做个参考。

概述

查看节点状态

 kubectl get nodes
NAME       STATUS     ROLES     AGE       VERSION
docker01   Ready      master    1y        v1.9.0
docker04   Ready      <none>    1y        v1.9.0
docker06   NotReady   <none>    1y        v1.9.0
docker08   Ready      <none>    89d       v1.9.0

居然有个节点处于NotReady 状态，不爽

那就执行下加入到集群的命令中看看是什么问题

切换到docker06 这个机子上

ssh root@docker06

运行加入集群命令

# kubeadm join --token 6be0d2.121fb2825cd41f64 192.168.100.61:6443 --discovery-token-ca-cert-hash sha256:4e671bcabdf9e35491c1e9b51ce06dc6900bdd5b53ad48a13419051b5f1382f6
[preflight] Running pre-flight checks.
	[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Some fatal errors occurred:
	[ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
	[ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
	[ERROR Swap]: running with swap on is not supported. Please disable swap

主要是两个问题

1：证书一存在

2：开启了虚拟存储 swap

第一步：关闭swap

vim /etc/fstab

注释掉SWAP分区项

#
# /etc/fstab
# Created by anaconda on Thu Apr  9 22:39:56 2020
#
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/mapper/cl-root     /                       xfs     defaults        0 0
UUID=e4976f5b-c44e-4fba-b0c7-3b10bb939db2 /boot                   ext4    defaults        1 2
/dev/mapper/cl-home     /home                   xfs     defaults        0 0
#/dev/mapper/cl-swap     swap                    swap    defaults        0 0

命令行执行

swapoff -a

第二步：重启node上的k8s

重启命令

kubeadm reset
[preflight] Running pre-flight checks.
[reset] Stopping the kubelet service.
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers.
[reset] No etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml". Assuming external etcd.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

第三步：再次执行加入集群的命令

 kubeadm join --token 6be0d2.121fb2825cd41f64 192.168.100.61:6443 --discovery-token-ca-cert-hash sha256:4e671bcabdf9e35491c1e9b51ce06dc6900bdd5b53ad48a13419051b5f1382f6
[preflight] Running pre-flight checks.
	[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "192.168.100.61:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.100.61:6443"
[discovery] Failed to connect to API Server "192.168.100.61:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "6be0d2" is invalid for this cluster, can't connect
[discovery] Trying to connect to API Server "192.168.100.61:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.100.61:6443"
[discovery] Failed to connect to API Server "192.168.100.61:6443": there is no JWS signed token in the cluster-info ConfigMap. This token id "6be0d2" is invalid for this cluster, can't connect
[discovery] Trying to connect to API Server "192.168.100.61:6443"

又是新问题，

原因是 k8s的token 有效期只有24小时，那就重新建一个token

第四步：新建一个token

kubeadm token create

要想永久token就用下面这个命令

kubeadm token create --ttl 0

查看token

# kubeadm token list
TOKEN                     TTL         EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
dxnj79.rnj561a137ri76ym   <invalid>   2018-11-02T14:06:43+08:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token
o4avtg.65ji6b778nyacw68   <forever>   <never>                     authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

第五步：获取ca证书sha256编码hash值

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
2cc3029123db737f234186636330e87b5510c173c669f513a9c0e0da395515b0

第六步：node节点加入

kubeadm join 10.167.11.153:6443 --token o4avtg.65ji6b778nyacw68 --discovery-token-ca-cert-hash sha256:2cc3029123db737f234186636330e87b5510c173c669f513a9c0e0da395515b0

kubeadm join 10.167.11.153:6443 --token o4avtg.65ji6b778nyacw68 --discovery-token-ca-cert-hash sha256:2cc3029123db737f234186636330e87b5510c173c669f513a9c0e0da395515b0

注意：红色部分对应上面2条命令的结果。

执行结果如下，表示已成功加入集群

[preflight] Running pre-flight checks.
	[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "192.168.100.61:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.100.61:6443"
[discovery] Requesting info from "https://192.168.100.61:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.100.61:6443"
[discovery] Successfully established connection with API Server "192.168.100.61:6443"

This node has joined the cluster:
* Certificate signing request was sent to master and a response
  was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

第七步：再看看进群的节点状态

]# kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
docker01   Ready     master    1y        v1.9.0
docker04   Ready     <none>    1y        v1.9.0
docker06   Ready     <none>    1y        v1.9.0
docker08   Ready     <none>    89d       v1.9.0

已经正常了。