使用docker方式安装etcd集群，带TLS证书

网上文档也多，安装的时候，还是踩了几个坑。

现在作一个安装记录吧。

1，先作自签名的证书ca-csr.json(为了和k8s共用根证书，可能将信息调为k8s)。

{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"ca": {
"expiry": "438000h"
},
"names": [
{
"C": "CN",
"ST": "ShangHai",
"L": "ShangHai",
"O": "kubernetes",
"OU": "system"
}
]
}

2,生成根证书和key。

cfssl gencert -initca
ca-csr.json | cfssljson -bare ca

这一步，会生成ca.pem和ca-key.pem文件。

3，有了根证书之后，建一个ca-config.json文件，用它来作为签发其它证书的配置文件。

{
"signing": {
"default": {
"expiry": "438000h"
},
"profiles": {
"server": {
"expiry": "438000h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "438000h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
},
"peer": {
"expiry": "438000h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}

4，生成一个etcd_server_demo.json。

{
"CN": "etcdServer",
"hosts": [
"127.0.0.1",
"localhost",
"etcd_ip_address"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"O": "etcd",
"OU": "etcd Security",
"C": "CN",
"L": "ShangHai",
"ST": "ShangHai"
}
]
}

这个json文件，我们暂时不生成证书，因为不同的etcd节点，需要证书认证的IP不一样，我们可以通过脚本把IP传进去，即时生成证书。

5，生成一个etcd_peer.json文件

{
"CN": "etcdPeer",
"hosts": [
"127.0.0.1",
"localhost",
　　　　　"peer*.ip",
"server.etcd.cluster.local",
"peer1.etcd.cluster.local",
"peer2.etcd.cluster.local",
"peer3.etcd.cluster.local",
"peer4.etcd.cluster.local",
"peer5.etcd.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"O": "etcd",
"OU": "etcd Security",
"C": "CN",
"L": "ShangHai",
"ST": "ShangHai"
}
]
}

6，使用上面的json文件，生成peer之间证书(未启用peer认证，因为是内部网，只启用了apiserver到2379的tls认证)。

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=etcd_ca-config.json -profile=peer etcd-peer.json | cfssljson -bare peer

 

7，生成一个etcd_client.json文件

{
"CN": "etcdClient",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"O": "etcd",
"OU": "etcd Security",
"C": "CN",
"L": "ShangHai",
"ST": "ShangHai"
}
]
}

 

8，使用上面的json文件，生成一个客户端认证的证书(这个证书，改了名字之后，可用于k8s的healthcheck，也可用于apiserver-etcd-client)。

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=etcd_ca-config.json -profile=client etcd-client.json | cfssljson -bare client

9,生成了这些证书之后，将所有证书和cfssl放到pik/目录下，使用下面这个etcd.sh脚本，进行etcd的集群安装。

#! /usr/bin/env bash
set -e
set -u
set -x
THIS_HOST=$1
REGISTRY=harbor/k8s.gcr.io/etcd-amd64
ETCD_VERSION=3.2.18
TOKEN=k8s-etcd-token
CLUSTER_STATE=new
HOST_1=1.2.3.4
HOST_2=5.6.7.8
HOST_3=9.0.1.2
CLUSTER=${HOST_1}=http://${HOST_1}:2380,${HOST_2}=http://${HOST_2}:2380,${HOST_3}=http://${HOST_3}:2380
DATA_DIR=/etcd/etcd-data
PKI_SRC_DIR=etcd/pki
PKI_DST_DIR=/pki
LOCAL_IP=$(hostname -I)
# 以上变量,三个节点相同
#判断输入的IP是否在集群内，是否为本机IP
if [[ ${THIS_HOST} != ${HOST_1} && ${THIS_HOST} != ${HOST_2} && ${THIS_HOST} != ${HOST_3} ]]; then
echo "ip not in the etcd cluster host."
exit 110
fi
if ! [[ ${LOCAL_IP} =~ ${THIS_HOST} ]]; then
echo "ip not in the local ip."
exit 110
fi
# 替换IP生成证书
/bin/cp -rf pki/etcd_server_demo.json pki/etcd_server.json
sed -i "s/etcd_ip_address/${THIS_HOST}/g" pki/etcd_server.json
pki/cfssl gencert -ca=pki/ca.pem -ca-key=pki/ca-key.pem -config=pki/ca-config.json -profile=server pki/etcd_server.json|pki/cfssljson -bare pki/server
# 运行docker
docker run 
-p 2379:2379 
-p 2380:2380 
--volume=${DATA_DIR}:/etcd-data 
--volume=${PKI_SRC_DIR}:${PKI_DST_DIR} 
--name etcd ${REGISTRY}:${ETCD_VERSION} 
/usr/local/bin/etcd 
--data-dir=/etcd-data --name ${THIS_HOST} 
--initial-advertise-peer-urls http://${THIS_HOST}:2380 
--listen-peer-urls http://0.0.0.0:2380 
--advertise-client-urls https://${THIS_HOST}:2379 
--listen-client-urls https://0.0.0.0:2379 
--initial-cluster ${CLUSTER} 
--initial-cluster-state ${CLUSTER_STATE} 
--initial-cluster-token ${TOKEN} 
--cert-file=${PKI_DST_DIR}/server.pem 
--key-file=${PKI_DST_DIR}/server-key.pem 
--trusted-ca-file=${PKI_DST_DIR}/ca.pem

 

10，安装集群。将etcd.sh里的HOST_1~3更改为计划集群的IP地址。使用，在每台机器上，使用sh etcd.sh [本地IP]，即可安装。

11，常用的测试集群和测试集群的命令，还带tls的。

./etcdctl --ca-file=pki/ca.pem --cert-file=pki/client.pem --key-file=pki/client-key.pem --endpoints=https://1.2.3.4:2379 cluster-health

./etcdctl --ca-file=pki/ca.pem --cert-file=pki/client.pem --key-file=pki/client-key.pem --endpoints=https://1.2.3.4:2379 member list

./etcdctl --ca-file=pki/ca.pem --cert-file=pki/client.pem --key-file=pki/client-key.pem --endpoints=https://1.2.3.4:2379 set /message hello

./etcdctl --ca-file=pki/ca.pem --cert-file=pki/client.pem --key-file=pki/client-key.pem --endpoints=https://1.2.3.4:2379 get /message

curl -k --cert pki/client.pem
--key pki/client-key.pem https://1.2.3.4:2379/v2/keys/message