etcd报错：mvcc: database space exceeded

etcd报错：mvcc: database space exceeded

问题来源：恢复生产环境的时候发现apiserver连etcd报错，etcd一直重启

原因分析：
etcd服务未设置自动压缩参数（auto-compact）
etcd 默认不会自动 compact，需要设置启动参数，或者通过命令进行compact，如果变更频繁建议设置，否则会导致空间和内存的浪费以及错误。etcd v3 的默认的 backend quota 2GB，如果不compact，boltdb 文件大小超过这个限制后，就会报错：”Error: etcdserver: mvcc: database space exceeded”，导致数据无法写入。

处理过程：
我这里的master节点ip是192.168.10.203，192.168.10.204，192.168.10.205

1 查看告警
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.203:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem alarm list
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.204:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem alarm list
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.205:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem alarm list

2 获取旧版本号
[root@ ~]# rev=$(/opt/k8s/bin/etcdctl --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=“https://127.0.0.1:2379” endpoint status --write-out=“json” |egrep -o ‘“revision”:[0-9]’|egrep -o '[0-9].’)
[root@ ~]# echo $rev
846418475

3 整合压缩旧版本数据
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.203:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem compact $rev
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.204:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem compact $rev
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.205:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem compact $rev

4 执行碎片整理
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.203:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem defrag
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.204:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem defrag
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.205:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem defrag

5 解除告警
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.203:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem alarm disarm
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.204:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem alarm disarm
[root@ ~]# /opt/k8s/bin/etcdctl --endpoints=https://192.168.10.205:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem alarm disarm

6 备份以及查看备份数据信息
ETCDCTL_API=3 etcdctl snapshot save backup.db
ETCDCTL_API=3 etcdctl snapshot status backup.db

最后

以上就是健忘胡萝卜为你收集整理的etcd报错：mvcc: database space exceeded的全部内容，希望文章能够帮你解决etcd报错：mvcc: database space exceeded所遇到的程序开发问题。

如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。