「Kubernetes」- 搭建高可用集群 @20210331

# 添加内核模块
cat >> /etc/modules-load.d/k8s.conf <<EOF
br_netfilter
# run kube-proxy in ipvs mode
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
EOF
systemctl restart systemd-modules-load.service
# 修改内核参数
cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

# 添加内核模块
cat >> /etc/modules-load.d/k8s.conf <<EOF
# run kube-proxy in ipvs mode
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4
EOF
systemctl restart systemd-modules-load.service

# **追加** Calico 要求的内核模块
# 那些注释（#）的内核模块，虽然官方文档要求，但是可能因为文档未更新。在 Ubuntu 18.04 TLS 中，我们未找到这些模块
# https://github.com/kubernetes-sigs/kubespray/issues/6289
cat >> /etc/modules-load.d/k8s.conf <<EOF
# for Calico
ip_set
ip_tables
ip6_tables
ipt_REJECT
ipt_rpfilter
ipt_set
nf_conntrack_netlink
# nf_conntrack_proto_sctp
sctp
xt_addrtype
xt_comment
xt_conntrack
# xt_icmp
# xt_icmp6
xt_ipvs
xt_mark
xt_multiport
# xt_rpfilter
xt_sctp
xt_set
xt_u32
ipip
EOF
# 加载内核模块配置
systemctl restart systemd-modules-load.service

sed -i -E 's%(^[^#].+sswaps.+)%# 1%g' /etc/fstab
swapoff -a

cat <<EOF | tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
systemctl enable docker
systemctl daemon-reload
systemctl restart docker

apt-get update 
&& apt-get install -y apt-transport-https ca-certificates curl
# 尽管下载 apt-key.gpg 存在困难，但是应该尽量从官方站点下载（请勿随意使用第三方密钥）
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat > /etc/apt/sources.list.d/kubernetes.list <<EOF
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update
# 安装工具
apt-get install -y kubelet=1.20.4-00 kubeadm=1.20.4-00 kubectl=1.20.4-00
apt-mark hold kubelet kubeadm kubectl # 禁止更新

cat >> /etc/hosts <<EOF
172.31.253.60 control-plane-endpoint.d3rm.org
EOF

mkdir -pv /etc/kube-vip/
cat > /etc/kube-vip/config.yaml <<EOF
localPeer:
id: k8s-cp-01
address: 172.31.253.61
port: 10000
remotePeers:
- id: k8s-cp-02
address: 172.31.253.62
port: 10000
- id: k8s-cp-03
address: 172.31.253.63
port: 10000
vip: 172.31.253.60
gratuitousARP: true
singleNode: false
startAsLeader: true
interface: "enp1s0"
loadBalancers:
- name: API Server Load Balancer
type: tcp
port: 8443
bindToVip: true
backends:
- port: 6443
address: 172.31.253.61
- port: 6443
address: 172.31.253.62
- port: 6443
address: 172.31.253.63
EOF

mkdir -pv /etc/kubernetes/manifests/
docker run -it --rm plndr/kube-vip:0.3.3 sample manifest 
> /etc/kubernetes/manifests/kube-vip.yaml

# kubeadm init 
--control-plane-endpoint "control-plane-endpoint.d3rm.org:8443"

--apiserver-bind-port 6443 
--image-repository "registry.aliyuncs.com/google_containers"
[init] Using Kubernetes version: v1.20.5
[preflight] Running pre-flight checks
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join control-plane-endpoint.d3rm.org:6443 --token lgm5el.esqwyahynz6eqyjx 
--discovery-token-ca-cert-hash sha256:52626a3d6bdab90344575f3eb3a3f03b3bc43d0ca4f0411781ae2cdea024b520 
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join control-plane-endpoint.d3rm.org:6443 --token lgm5el.esqwyahynz6eqyjx 
--discovery-token-ca-cert-hash sha256:52626a3d6bdab90344575f3eb3a3f03b3bc43d0ca4f0411781ae2cdea024b520

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

10541 kubelet.go:2184] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

# 按照文档的描述，我们的场景应该使用 Kubernetes API datastore 存储，且小于 50 个节点数
curl https://docs.projectcalico.org/manifests/calico.yaml -O

kubectl apply -f calico.yaml

# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973
# kubeadm token create --print-join-command --certificate-key 67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973
kubeadm join control-plane-endpoint.d3rm.org:6443 --token i4lcbu.5jpmipmdp6wjde3j
--discovery-token-ca-cert-hash sha256:95147bd44e3109a96414ac7f9d480ef5d42a43819bd91d22c84289f71b460bde
--control-plane --certificate-key 67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973
# kubeadm join control-plane-endpoint.d3rm.org:6443 --token i4lcbu.5jpmipmdp6wjde3j 
--discovery-token-ca-cert-hash sha256:95147bd44e3109a96414ac7f9d480ef5d42a43819bd91d22c84289f71b460bde 
--control-plane 
--certificate-key 67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973

kubeadm join control-plane-endpoint.d3rm.org:6443 --token 4pedgf.a2uc2vvtrknce1wb 
--discovery-token-ca-cert-hash sha256:0fce70ebfdc980066ea08cdfdc54a835215077834b54f10a36196c22a9dacb09

kubectl drain "<node name>" --delete-local-data --force --ignore-daemonsets #
kubeadm reset # **NOTE** run on <node name>
kubectl delete node "<node name>" #

# 获取 TOKEN 参数
kubeadm token list
# 查看已有 TOKEN
kubeadm token create # 创建新的 TOKEN
# 获取 discovery-token-ca-cert-hash 参数
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | 
openssl dgst -sha256 -hex | sed 's/^.* //'
# 加入集群
kubeadm join "control-plane-endpoint.d3rm.org:8443" --token "4pedgf.a2uc2vvtrknce1wb" 
--discovery-token-ca-cert-hash "sha256:0fce70ebfdc980066ea08cdfdc54a835215077834b54f10a36196c22a9dacb09"

# 获取 certificate key 参数
kubeadm init phase upload-certs --upload-certs
# 获取加入集群的命令
kubeadm token create --print-join-command --certificate-key "<certificate key>"
# 执行输出的 kubeadm join 命令

# kubeadm join control-plane-endpoint.d3rm.org:8443 --token zb93u8.0xbcfu00xlo8ulbr 
>
--discovery-token-ca-cert-hash sha256:396025ff1affbf913c70af034e20ab7c0385631625439db833a099eee88fa0a3 
>
--control-plane
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
error execution phase preflight:
One or more conditions for hosting a new control plane instance is not satisfied.
failure loading certificate for CA: couldn't load the certificate file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory
Please ensure that:
* The cluster has a stable controlPlaneEndpoint address.
* The certificates that must be shared among control plane instances are provided.
To see the stack trace of this error execute with --v=5 or higher

# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973
# kubeadm token create --print-join-command --certificate-key 67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973
kubeadm join control-plane-endpoint.d3rm.org:6443 --token i4lcbu.5jpmipmdp6wjde3j
--discovery-token-ca-cert-hash sha256:95147bd44e3109a96414ac7f9d480ef5d42a43819bd91d22c84289f71b460bde
--control-plane --certificate-key 67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973

# kubeadm join control-plane-endpoint.d3rm.org:6443 --token f16qw8.3sawd6jbv3eaytju 
--discovery-token-ca-cert-hash sha256:95147bd44e3109a96414ac7f9d480ef5d42a43819bd91d22c84289f71b460bde 
--control-plane 
--certificate-key e6f755259360e408894fde6a3eba4eebf889823fd06d59a2d555c061a03b62a8
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
error execution phase control-plane-prepare/download-certs: error downloading certs: error downloading the secret: secrets "kubeadm-certs" is forbidden: User "system:bootstrap:f16qw8" cannot get resource "secrets" in API group "" in the namespace "kube-system"
To see the stack trace of this error execute with --v=5 or higher

# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973
# kubeadm token create --print-join-command --certificate-key 67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973
kubeadm join control-plane-endpoint.d3rm.org:6443 --token i4lcbu.5jpmipmdp6wjde3j
--discovery-token-ca-cert-hash sha256:95147bd44e3109a96414ac7f9d480ef5d42a43819bd91d22c84289f71b460bde
--control-plane --certificate-key 67b240dd68a9f4dba7017164bb095b08cf21b6445e20199d529269b7bd685973