我是靠谱客的博主 满意缘分，最近开发中收集的这篇文章主要介绍kubernetes安装及集群搭建k8s安装安装环境安装步骤ssh设置node1ssh免密方便传输文件node1传输hosts文件更改主机名node1node2node3关闭防火墙沙盒kubernetes安装环境要求验证每个节点的mac地址和product_uuid是唯一的node1node2node3关闭swap分区查看是否关闭下载第三方依赖的库文件（会解决很多依赖关系）node1如果发现那两条显示没有那个文件或目录，然后先导入两个模块（多敲几次）让参数生效传输过去node2node3进入上面，觉得挺不错的，现在分享给大家，希望可以做个参考。
概述
k8s安装
kubernetes官网
安装环境
不支持centos8的系统
| IP | 服务 | 硬件要求 |
|---|---|---|
| 192.168.2.10（node1） | Docker（已安装）、kubernetes | 内存4G，双核CPU |
| 192.168.2.20（node2） | Docker（已安装）、kubernetes | 内存4G，双核CPU |
| 192.168.2.30（node3） | Docker（已安装）、kubernetes | 内存4G，双核CPU |
安装步骤
环境准备
为了修改文件传输文件的方便，将主机名更改为安装环境所述的node，并写入hosts文件
ssh设置
node1
[root@localhost ~]# vim /etc/hosts
192.168.2.10 node1
192.168.2.20 node2
192.168.2.30 node3
ssh免密
方便传输文件
node1
[root@localhost ~]# ssh-keygen
[root@localhost ~]# ssh-copy-id -i root@node2
[root@localhost ~]# ssh-copy-id -i root@node3
传输hosts文件
[root@localhost ~]# scp /etc/hosts root@node2:/etc
[root@localhost ~]# scp /etc/hosts root@node3:/etc
更改主机名
node1
[root@localhost ~]# hostname node1
[root@localhost ~]# bash
[root@node1 ~]#
node2
[root@localhost ~]# hostname node2
[root@localhost ~]# bash
[root@node2 ~]#
node3
[root@localhost ~]# hostname node3
[root@localhost ~]# bash
[root@node3 ~]#
关闭防火墙沙盒
node1/2/3
systemctl stop firewalld && systemctl disable firewalld
setenforce 0
vim /etc/selinux/config
# 修改
SELINUX=disabled
kubernetes安装环境要求
官方文档
防火墙端口
如果不关闭防火墙，请放下以下端口
Master节点
Node节点
验证每个节点的mac地址和product_uuid是唯一的
kubernetes通过这两个值来确定集群中的节点
node1
[root@node1 ~]# cat /sys/class/dmi/id/product_uuid
19D84D56-03C0-B76A-96EA-37D54065C278
node2
[root@node2 ~]# cat /sys/class/dmi/id/product_uuid
D8304D56-8AB4-7444-D531-C67AF37581C7
node3
[root@node3 ~]# cat /sys/class/dmi/id/product_uuid
69E84D56-2D4B-3E4E-9DE8-4D7B44052C25
关闭swap分区
三台操作同样，这一步不要scp，因为fstab不一样
swapoff -a
vim /etc/fstab
将分区类型为swap的一行注释掉
/dev/mapper/centos-swap swap
查看是否关闭
[root@node1 ~]# free -m
total used free shared buff/cache available
Mem: 2793 1024 206 14 1562 1334
Swap: 0 0 0
下载第三方依赖的库文件（会解决很多依赖关系）
node1/2/3
yum install epel-release -y
node1
[root@node1 ~]# vim /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0 # 当内存oom是不使用交换分区
如果发现那两条显示没有那个文件或目录，然后先导入两个模块（多敲几次）
modprobe ip_vs_rr
modprobe br_netfilter
让参数生效
[root@node1 ~]# sysctl -p
net.ipv4.ip_forward = 1
vm.swappiness = 0
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
传输过去
[root@node1 ~]# scp /etc/sysctl.conf root@node2:/etc
[root@node1 ~]# scp /etc/sysctl.conf root@node3:/etc
node2
[root@node2 ~]# modprobe ip_vs_rr
[root@node2 ~]# modprobe br_netfilter
[root@node2 ~]# sysctl -p
node3
[root@node3 ~]# modprobe ip_vs_rr
[root@node3 ~]# modprobe br_netfilter
[root@node3 ~]# sysctl -p
kubernetes安装
这里使用阿里云镜像站安装
kubernetes阿里云镜像站
node1
进入上面这个页面，使用centos的yum源
[root@node1 ~]# vim /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
将yum源传给每台主机
[root@node1 ~]# scp /etc/yum.repos.d/kubernetes.repo root@node2:/etc/yum.repos.d/
[root@node1 ~]# scp /etc/yum.repos.d/kubernetes.repo root@node3:/etc/yum.repos.d/
node1/2/3
yum install -y kubelet kubeadm kubectl
- kubeadm：引导集群的命令。
- kubelet：在群集中所有计算机上运行的组件，它执行诸如启动Pod和容器之类的操作。
- kubectl：用于与您的集群通信的命令行工具。
三台启动服务
systemctl enable kubelet.service && systemctl start kubelet.service
systemctl start docker.service
安装kubernetes的tab快捷键
[root@node1 ~]# source <(kubectl completion bash)
[root@node1 ~]# source <(kubeadm completion bash)
查看版本
[root@node1 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:49:29Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
kubernetes创建集群
node1
执行这条后会等待一段时间，需要进行下载镜像
kubeadm init --apiserver-advertise-address 192.168.2.10 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.18.3 --pod-network-cidr=10.244.0.0/16
参数释义
–apiserver-advertise-address：通告侦听地址
–image-repository：指定镜像地址使用阿里云的，默认会使用谷歌镜像
–kubernetes-version：指定当前的kubernetes的版本
–pod-network-cidr=10.244.0.0/16：flannel网络的固定地址范围
成功则会返回如下信息：
# 如果使用普通账号操作，执行以下三条
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 部署集群网络
https://kubernetes.io/docs/concepts/cluster-administration/addons/
# 其他kubernetes主机通过以下命令加入kubernetes集群
kubeadm join 192.168.2.10:6443 --token 1dnjl6.51yjfkjm2ys5p6gx
--discovery-token-ca-cert-hash sha256:e5b6bfc12b3b0e7101ffb6433ae0c0be7a7a24aaf322112b59e1b577c0f9256d
如果是root用户先执行以下命令
export KUBECONFIG=/etc/kubernetes/admin.conf
加入命令创建有效期24小时，允许管理我们的令牌，如果丢了就会出现错误
查看计算机当中的令牌
[root@node1 ~]# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
1dnjl6.51yjfkjm2ys5p6gx 17h 2020-06-11T09:43:35+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token
部署集群网络
浏览器打开地址https://kubernetes.io/docs/concepts/cluster-administration/addons/
找到下图位置，点击flannel
进入Documentation目录下
找到kube-flannel.yml文件
将里面的内容进行复制到虚拟机的kube-flannel.yml文件
[root@node1 ~]# vi kube-flannel.yml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-amd64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: kubernetes.io/arch
operator: In
values:
- amd64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.12.0-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: kubernetes.io/arch
operator: In
values:
- arm64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.12.0-arm64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-arm64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-arm
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: kubernetes.io/arch
operator: In
values:
- arm
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.12.0-arm
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-arm
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-ppc64le
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: kubernetes.io/arch
operator: In
values:
- ppc64le
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.12.0-ppc64le
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-ppc64le
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-s390x
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: kubernetes.io/arch
operator: In
values:
- s390x
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.12.0-s390x
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.12.0-s390x
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
找到该位置，此镜像需要使用docker下载
所有主机尽量都pull一下也是可以的
[root@node1 ~]# docker pull quay.io/coreos/flannel:v0.12.0-amd64
部署集群网络
[root@node1 ~]# kubectl apply -f kube-flannel.yml
node2/3加入集群
kubeadm join 192.168.2.10:6443 --token 1dnjl6.51yjfkjm2ys5p6gx
--discovery-token-ca-cert-hash sha256:e5b6bfc12b3b0e7101ffb6433ae0c0be7a7a24aaf322112b59e1b577c0f9256d
完成后输出一下信息
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
node1
在node1运行kubectl get nodes查看集群节点，状态的为Notready的是还没有准备好，多等一会，再去查看
[root@node1 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
server1 Ready master 6h35m v1.18.3
server2 Ready <none> 6h27m v1.18.3
server3 Ready <none> 6h27m v1.18.3
如果发现节点为Notready，时间长也不行，将默认的网络插件cni去掉，集群中全部修改
vim /var/lib/kubelet/kubeadm-flags.env
# 删除--network-plugin=cni
重新启动
systemctl daemon-reload
systemctl restart kubelet
此时的kubelet才启动成功。
重新查看集群状态
执行kubectl get pod --all-namespaces，查看所有命名空间的pod，如果有状态为PodInitializing，就多等一会，网络原因，pod插件没有下载好，多等等就好了，全部这台为running即可
[root@node1 ~]# kubectl get pod --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7ff77c879f-dvh7g 1/1 Running 0 6h38m
kube-system coredns-7ff77c879f-gk9zd 1/1 Running 0 6h38m
kube-system etcd-server1 1/1 Running 0 6h38m
kube-system kube-apiserver-server1 1/1 Running 0 6h38m
kube-system kube-controller-manager-server1 1/1 Running 0 6h38m
kube-system kube-flannel-ds-amd64-tdcnh 1/1 Running 0 6h30m
kube-system kube-flannel-ds-amd64-thsxp 1/1 Running 2 6h30m
kube-system kube-flannel-ds-amd64-twqjf 1/1 Running 2 6h30m
kube-system kube-proxy-4drcp 1/1 Running 0 6h30m
kube-system kube-proxy-fts9q 1/1 Running 0 6h38m
kube-system kube-proxy-jnb7c 1/1 Running 0 6h30m
kube-system kube-scheduler-server1 1/1 Running 0 6h38m
集群搭建成功
最后
以上就是满意缘分为你收集整理的kubernetes安装及集群搭建k8s安装安装环境安装步骤ssh设置node1ssh免密方便传输文件node1传输hosts文件更改主机名node1node2node3关闭防火墙沙盒kubernetes安装环境要求验证每个节点的mac地址和product_uuid是唯一的node1node2node3关闭swap分区查看是否关闭下载第三方依赖的库文件（会解决很多依赖关系）node1如果发现那两条显示没有那个文件或目录，然后先导入两个模块（多敲几次）让参数生效传输过去node2node3进入上面的全部内容，希望文章能够帮你解决kubernetes安装及集群搭建k8s安装安装环境安装步骤ssh设置node1ssh免密方便传输文件node1传输hosts文件更改主机名node1node2node3关闭防火墙沙盒kubernetes安装环境要求验证每个节点的mac地址和product_uuid是唯一的node1node2node3关闭swap分区查看是否关闭下载第三方依赖的库文件（会解决很多依赖关系）node1如果发现那两条显示没有那个文件或目录，然后先导入两个模块（多敲几次）让参数生效传输过去node2node3进入上面所遇到的程序开发问题。
如果觉得靠谱客网站的内容还不错，欢迎将靠谱客网站推荐给程序员好友。
本图文内容来源于网友提供，作为学习参考使用，或来自网络收集整理，版权属于原作者所有。
发表评论 取消回复